[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:41:08 dist-test-slave-l31f krb5kdc[796](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:41:08 dist-test-slave-l31f krb5kdc[796](info): set up 2 sockets
Sep 29 16:41:08 dist-test-slave-l31f krb5kdc[796](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260929 16:41:09.430310 789 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.009s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/krb5kdc/test-user.keytab.
Sep 29 16:41:09 dist-test-slave-l31f krb5kdc[796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:41:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:41:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.197.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
I20260929 16:41:09.626169 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:33837
--webserver_interface=127.0.197.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:35357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.126:33837
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:41:09.993157 809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:41:09.993572 809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:41:09.993757 809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:41:10.001401 809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:41:10.001616 809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:41:10.001735 809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:41:10.001842 809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:41:10.001914 809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:41:10.012828 809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:35357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.197.126:33837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:33837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.197.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:41:10.014340 809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:41:10.015872 809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:41:10.030777 814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:10.031487 817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:10.031939 815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:41:10.033397 809 server_base.cc:1034] running on GCE node
I20260929 16:41:10.034384 809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:41:10.036160 809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:41:10.037398 809 hybrid_clock.cc:648] HybridClock initialized: now 1790700070037311 us; error 85 us; skew 500 ppm
I20260929 16:41:10.037849 809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:41:10 dist-test-slave-l31f krb5kdc[796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700070, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:41:10.041422 809 init.cc:377] Logged in from keytab as kudu/127.0.197.126@KRBTEST.COM (short username kudu)
I20260929 16:41:10.043390 809 webserver.cc:460] Webserver started at http://127.0.197.126:45025/ using document root <none> and password file <none>
I20260929 16:41:10.044183 809 fs_manager.cc:359] Metadata directory not provided
I20260929 16:41:10.044301 809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:41:10.044693 809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:41:10.048120 809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/master-0/data/instance:
uuid: "b3846a30980045a2b858647b91bc223c"
format_stamp: "Formatted at 2026-09-29 16:41:10 on dist-test-slave-l31f"
server_key: "eacd9b9dfd292ff0aecd6c76aa905117"
server_key_iv: "ec4cdb5faabf9aa4903a25e695b0cb6f"
server_key_version: "encryptionkey@0"
I20260929 16:41:10.049068 809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/master-0/wal/instance:
uuid: "b3846a30980045a2b858647b91bc223c"
format_stamp: "Formatted at 2026-09-29 16:41:10 on dist-test-slave-l31f"
server_key: "eacd9b9dfd292ff0aecd6c76aa905117"
server_key_iv: "ec4cdb5faabf9aa4903a25e695b0cb6f"
server_key_version: "encryptionkey@0"
I20260929 16:41:10.056666 809 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260929 16:41:10.061012 824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:10.062870 809 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260929 16:41:10.063407 809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "b3846a30980045a2b858647b91bc223c"
format_stamp: "Formatted at 2026-09-29 16:41:10 on dist-test-slave-l31f"
server_key: "eacd9b9dfd292ff0aecd6c76aa905117"
server_key_iv: "ec4cdb5faabf9aa4903a25e695b0cb6f"
server_key_version: "encryptionkey@0"
I20260929 16:41:10.063680 809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:41:10.078641 809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:41:10.083803 809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:41:10.084188 809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:41:10.108304 809 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.126:33837
I20260929 16:41:10.108322 875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.126:33837 every 8 connection(s)
I20260929 16:41:10.109939 809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:41:10.114146 876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:41:10.117069 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 809
I20260929 16:41:10.117429 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/master-0/wal/instance
I20260929 16:41:10.120116 789 external_mini_cluster.cc:1428] Setting key c0e7b1b7d70305da84e7465c80ba7b3d
I20260929 16:41:10.129895 876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3846a30980045a2b858647b91bc223c: Bootstrap starting.
I20260929 16:41:10.133813 876 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3846a30980045a2b858647b91bc223c: Neither blocks nor log segments found. Creating new log.
I20260929 16:41:10.134979 876 log.cc:824] T 00000000000000000000000000000000 P b3846a30980045a2b858647b91bc223c: Log is configured to *not* fsync() on all Append() calls
I20260929 16:41:10.138517 876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3846a30980045a2b858647b91bc223c: No bootstrap required, opened a new log
I20260929 16:41:10.146880 876 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3846a30980045a2b858647b91bc223c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3846a30980045a2b858647b91bc223c" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33837 } }
I20260929 16:41:10.147447 876 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3846a30980045a2b858647b91bc223c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:41:10.147584 876 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3846a30980045a2b858647b91bc223c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3846a30980045a2b858647b91bc223c, State: Initialized, Role: FOLLOWER
I20260929 16:41:10.148343 876 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3846a30980045a2b858647b91bc223c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3846a30980045a2b858647b91bc223c" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33837 } }
I20260929 16:41:10.148669 876 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3846a30980045a2b858647b91bc223c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:41:10.148818 876 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3846a30980045a2b858647b91bc223c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:41:10.149014 876 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3846a30980045a2b858647b91bc223c [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:41:10.151453 876 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3846a30980045a2b858647b91bc223c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3846a30980045a2b858647b91bc223c" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33837 } }
I20260929 16:41:10.151990 876 leader_election.cc:302] T 00000000000000000000000000000000 P b3846a30980045a2b858647b91bc223c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3846a30980045a2b858647b91bc223c; no voters:
I20260929 16:41:10.152559 876 leader_election.cc:288] T 00000000000000000000000000000000 P b3846a30980045a2b858647b91bc223c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:41:10.152724 881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3846a30980045a2b858647b91bc223c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:41:10.153501 881 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3846a30980045a2b858647b91bc223c [term 1 LEADER]: Becoming Leader. State: Replica: b3846a30980045a2b858647b91bc223c, State: Running, Role: LEADER
I20260929 16:41:10.154156 881 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3846a30980045a2b858647b91bc223c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3846a30980045a2b858647b91bc223c" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33837 } }
I20260929 16:41:10.155282 876 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3846a30980045a2b858647b91bc223c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:41:10.157612 883 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3846a30980045a2b858647b91bc223c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3846a30980045a2b858647b91bc223c. Latest consensus state: current_term: 1 leader_uuid: "b3846a30980045a2b858647b91bc223c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3846a30980045a2b858647b91bc223c" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33837 } } }
I20260929 16:41:10.158061 883 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3846a30980045a2b858647b91bc223c [sys.catalog]: This master's current role is: LEADER
Sep 29 16:41:10 dist-test-slave-l31f krb5kdc[796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:41:10.162331 882 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3846a30980045a2b858647b91bc223c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3846a30980045a2b858647b91bc223c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3846a30980045a2b858647b91bc223c" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33837 } } }
I20260929 16:41:10.162736 882 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3846a30980045a2b858647b91bc223c [sys.catalog]: This master's current role is: LEADER
I20260929 16:41:10.167243 886 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:41:10.175465 886 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:41:10.176875 879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:10.125982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43316 (local address 127.0.197.126:33837)
0929 16:41:10.128416 (+ 2434us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:10.128433 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:10.128495 (+ 62us) server_negotiation.cc:407] Connection header received
0929 16:41:10.138228 (+ 9733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:10.138252 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:10.138779 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:10.139322 (+ 543us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:10.142564 (+ 3242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:10.144485 (+ 1921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:10.146449 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:10.147280 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:10.147561 (+ 281us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:10.161321 (+ 13760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:10.161334 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:10.161357 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:10.161425 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:10.167316 (+ 5891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:10.169340 (+ 2024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:10.169357 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:10.169369 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:10.169699 (+ 330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:10.170423 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:10.170429 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:10.170432 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:10.171270 (+ 838us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:10.171616 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:10.174527 (+ 2911us) server_negotiation.cc:299] Negotiation successful
0929 16:41:10.175161 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2165,"thread_start_us":1881,"threads_started":1}
I20260929 16:41:10.189198 886 catalog_manager.cc:1347] Generated new cluster ID: 1bb91e969cb04dbe97dc1fc61c97c513
I20260929 16:41:10.189338 886 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:41:10.215777 886 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:41:10.217490 886 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:41:10.225826 886 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3846a30980045a2b858647b91bc223c: Generated new TSK 0
I20260929 16:41:10.226827 886 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.197.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab.
Entry for principal kudu/127.0.197.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab.
I20260929 16:41:10.284651 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.65:0
--local_ip_for_outbound_sockets=127.0.197.65
--webserver_interface=127.0.197.65
--webserver_port=0
--tserver_master_addrs=127.0.197.126:33837
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab
--principal=kudu/127.0.197.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:35357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:41:10.654402 902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:41:10.654778 902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:41:10.654958 902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:41:10.662791 902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:41:10.662916 902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:41:10.663224 902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.65
I20260929 16:41:10.675027 902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:35357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab
--principal=kudu/127.0.197.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.197.65
--webserver_port=0
--tserver_master_addrs=127.0.197.126:33837
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.65
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:41:10.676469 902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:41:10.677913 902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:41:10.692083 910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:10.692268 908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:10.692230 907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:41:10.693463 902 server_base.cc:1034] running on GCE node
I20260929 16:41:10.694151 902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:41:10.695281 902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:41:10.696501 902 hybrid_clock.cc:648] HybridClock initialized: now 1790700070696458 us; error 52 us; skew 500 ppm
I20260929 16:41:10.696856 902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:41:10 dist-test-slave-l31f krb5kdc[796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700070, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:41:10.700719 902 init.cc:377] Logged in from keytab as kudu/127.0.197.65@KRBTEST.COM (short username kudu)
I20260929 16:41:10.702450 902 webserver.cc:460] Webserver started at http://127.0.197.65:42107/ using document root <none> and password file <none>
I20260929 16:41:10.703673 902 fs_manager.cc:359] Metadata directory not provided
I20260929 16:41:10.703887 902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:41:10.704486 902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:41:10.707948 902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/ts-0/data/instance:
uuid: "319fe8cd823b491b9d2e6f6fda1d7f98"
format_stamp: "Formatted at 2026-09-29 16:41:10 on dist-test-slave-l31f"
server_key: "73b39356d51c52de6f76cc404b7b85b7"
server_key_iv: "298755ab764b6d38e4b3148ff19a05c7"
server_key_version: "encryptionkey@0"
I20260929 16:41:10.708799 902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/ts-0/wal/instance:
uuid: "319fe8cd823b491b9d2e6f6fda1d7f98"
format_stamp: "Formatted at 2026-09-29 16:41:10 on dist-test-slave-l31f"
server_key: "73b39356d51c52de6f76cc404b7b85b7"
server_key_iv: "298755ab764b6d38e4b3148ff19a05c7"
server_key_version: "encryptionkey@0"
I20260929 16:41:10.714660 902 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260929 16:41:10.718928 917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:10.720715 902 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260929 16:41:10.720959 902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/ts-0/wal
uuid: "319fe8cd823b491b9d2e6f6fda1d7f98"
format_stamp: "Formatted at 2026-09-29 16:41:10 on dist-test-slave-l31f"
server_key: "73b39356d51c52de6f76cc404b7b85b7"
server_key_iv: "298755ab764b6d38e4b3148ff19a05c7"
server_key_version: "encryptionkey@0"
I20260929 16:41:10.721220 902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:41:10.745038 902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:41:10.750255 902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:41:10.750603 902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:41:10.751808 902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:41:10.753321 902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:41:10.753443 902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:10.753600 902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:41:10.753676 902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:10.795431 902 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.65:44385
I20260929 16:41:10.795459 1029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.65:44385 every 8 connection(s)
I20260929 16:41:10.796952 902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
I20260929 16:41:10.806273 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 902
I20260929 16:41:10.806541 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/ts-0/wal/instance
I20260929 16:41:10.807410 789 external_mini_cluster.cc:1428] Setting key 5999b97cff3678f4455ce66a6151af9d
Sep 29 16:41:10 dist-test-slave-l31f krb5kdc[796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700070, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.65@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:41:10.821702 1032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:10.800038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:39819 (local address 127.0.197.126:33837)
0929 16:41:10.800544 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:10.800551 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:10.801705 (+ 1154us) server_negotiation.cc:407] Connection header received
0929 16:41:10.804313 (+ 2608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:10.804318 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:10.804401 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:10.804519 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:10.806361 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:10.807598 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:10.809614 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:10.810031 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:10.810253 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:10.815217 (+ 4964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:10.815223 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:10.815227 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:10.815265 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:10.817969 (+ 2704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:10.818941 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:10.818955 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:10.818958 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:10.819174 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:10.819665 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:10.819670 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:10.819674 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:10.819971 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:10.820145 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:10.821065 (+ 920us) server_negotiation.cc:299] Negotiation successful
0929 16:41:10.821316 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":248,"threads_started":1}
I20260929 16:41:10.822798 1033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:10.800750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:33837 (local address 127.0.197.65:39819)
0929 16:41:10.801506 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:10.801550 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:10.803877 (+ 2327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:10.804801 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:10.804810 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:10.805409 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:41:10.806101 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:10.806122 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:10.807797 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:10.807802 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:10.809454 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:10.809469 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:10.810362 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:10.810372 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:10.810632 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:10.811902 (+ 1270us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:41:10.811948 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:41:10.814878 (+ 2930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:41:10.818229 (+ 3351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:10.818241 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:10.818271 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:10.818781 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:10.819308 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:10.819313 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:10.819317 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:10.819534 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:10.820118 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:41:10.820127 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:41:10.820628 (+ 501us) client_negotiation.cc:769] Sending connection context
0929 16:41:10.820960 (+ 332us) client_negotiation.cc:240] Negotiation successful
0929 16:41:10.821568 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":301,"threads_started":1}
WARNING: no policy specified for kudu/127.0.197.66@KRBTEST.COM; defaulting to no policy
I20260929 16:41:10.824812 1030 heartbeater.cc:347] Connected to a master server at 127.0.197.126:33837
I20260929 16:41:10.825222 1030 heartbeater.cc:464] Registering TS with master...
I20260929 16:41:10.826133 1030 heartbeater.cc:511] Master 127.0.197.126:33837 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.66@KRBTEST.COM" created.
I20260929 16:41:10.829402 841 ts_manager.cc:194] Registered new tserver with Master: 319fe8cd823b491b9d2e6f6fda1d7f98 (127.0.197.65:44385)
I20260929 16:41:10.831784 841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.65@KRBTEST.COM'} at 127.0.197.65:39819
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab.
Entry for principal kudu/127.0.197.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab.
I20260929 16:41:10.856693 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.66:0
--local_ip_for_outbound_sockets=127.0.197.66
--webserver_interface=127.0.197.66
--webserver_port=0
--tserver_master_addrs=127.0.197.126:33837
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab
--principal=kudu/127.0.197.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:35357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:41:11.228523 1036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:41:11.228837 1036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:41:11.228937 1036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:41:11.237047 1036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:41:11.237197 1036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:41:11.237319 1036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.66
I20260929 16:41:11.247629 1036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:35357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab
--principal=kudu/127.0.197.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.197.66
--webserver_port=0
--tserver_master_addrs=127.0.197.126:33837
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.66
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:41:11.248984 1036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:41:11.250439 1036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:41:11.265925 1044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:11.265884 1042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:11.265952 1041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:41:11.267289 1036 server_base.cc:1034] running on GCE node
I20260929 16:41:11.268117 1036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:41:11.269527 1036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:41:11.270758 1036 hybrid_clock.cc:648] HybridClock initialized: now 1790700071270686 us; error 68 us; skew 500 ppm
I20260929 16:41:11.271183 1036 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:41:11 dist-test-slave-l31f krb5kdc[796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700071, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:41:11.275274 1036 init.cc:377] Logged in from keytab as kudu/127.0.197.66@KRBTEST.COM (short username kudu)
I20260929 16:41:11.277737 1036 webserver.cc:460] Webserver started at http://127.0.197.66:37197/ using document root <none> and password file <none>
I20260929 16:41:11.278612 1036 fs_manager.cc:359] Metadata directory not provided
I20260929 16:41:11.278780 1036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:41:11.279215 1036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:41:11.282972 1036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/ts-1/data/instance:
uuid: "a3a6dd4cf52b44e6a9d4c509d4b8a0bf"
format_stamp: "Formatted at 2026-09-29 16:41:11 on dist-test-slave-l31f"
server_key: "9fdf8699d043bc6d5fdc82fae4d5cbbb"
server_key_iv: "21e8988c837a5179832652c4661b86b3"
server_key_version: "encryptionkey@0"
I20260929 16:41:11.284065 1036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/ts-1/wal/instance:
uuid: "a3a6dd4cf52b44e6a9d4c509d4b8a0bf"
format_stamp: "Formatted at 2026-09-29 16:41:11 on dist-test-slave-l31f"
server_key: "9fdf8699d043bc6d5fdc82fae4d5cbbb"
server_key_iv: "21e8988c837a5179832652c4661b86b3"
server_key_version: "encryptionkey@0"
I20260929 16:41:11.290537 1036 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260929 16:41:11.295313 1051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:11.296860 1036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:41:11.297087 1036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/ts-1/wal
uuid: "a3a6dd4cf52b44e6a9d4c509d4b8a0bf"
format_stamp: "Formatted at 2026-09-29 16:41:11 on dist-test-slave-l31f"
server_key: "9fdf8699d043bc6d5fdc82fae4d5cbbb"
server_key_iv: "21e8988c837a5179832652c4661b86b3"
server_key_version: "encryptionkey@0"
I20260929 16:41:11.297322 1036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:41:11.319906 1036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:41:11.325449 1036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:41:11.325794 1036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:41:11.326929 1036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:41:11.328397 1036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:41:11.328477 1036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:11.328670 1036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:41:11.328723 1036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:11.371320 1036 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.66:40843
I20260929 16:41:11.371352 1163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.66:40843 every 8 connection(s)
I20260929 16:41:11.372783 1036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
I20260929 16:41:11.378258 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 1036
I20260929 16:41:11.378490 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/ts-1/wal/instance
I20260929 16:41:11.378916 789 external_mini_cluster.cc:1428] Setting key b5f5acb3fa69964775f6a8d0ceffe191
Sep 29 16:41:11 dist-test-slave-l31f krb5kdc[796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700071, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.66@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.197.67@KRBTEST.COM; defaulting to no policy
I20260929 16:41:11.395651 1166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:11.375697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:36399 (local address 127.0.197.126:33837)
0929 16:41:11.376200 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:11.376205 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:11.377307 (+ 1102us) server_negotiation.cc:407] Connection header received
0929 16:41:11.379424 (+ 2117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:11.379430 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:11.379533 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:11.379667 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:11.381630 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:11.382347 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:11.384238 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:11.384696 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:11.384842 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:11.389310 (+ 4468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:11.389316 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:11.389320 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:11.389357 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:11.392215 (+ 2858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:11.393127 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:11.393139 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:11.393142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:11.393279 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:11.393763 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:11.393767 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:11.393769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:11.394045 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:11.394208 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:11.394976 (+ 768us) server_negotiation.cc:299] Negotiation successful
0929 16:41:11.395259 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":231,"threads_started":1}
I20260929 16:41:11.396826 1167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:11.376347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:33837 (local address 127.0.197.66:36399)
0929 16:41:11.377116 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:11.377166 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:11.379111 (+ 1945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:11.379941 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:11.379951 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:11.380776 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:41:11.381426 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:11.381446 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:11.382524 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:11.382529 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:11.384040 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:11.384057 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:11.384970 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:11.384977 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:11.385240 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:11.385992 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:41:11.386023 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:41:11.388978 (+ 2955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:41:11.392424 (+ 3446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:11.392435 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:11.392466 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:11.392960 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:11.393406 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:11.393411 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:11.393415 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:11.393630 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:11.394181 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:41:11.394189 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:41:11.394580 (+ 391us) client_negotiation.cc:769] Sending connection context
0929 16:41:11.394866 (+ 286us) client_negotiation.cc:240] Negotiation successful
0929 16:41:11.395501 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":297,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.67@KRBTEST.COM" created.
I20260929 16:41:11.398758 1164 heartbeater.cc:347] Connected to a master server at 127.0.197.126:33837
I20260929 16:41:11.399489 1164 heartbeater.cc:464] Registering TS with master...
I20260929 16:41:11.400586 1164 heartbeater.cc:511] Master 127.0.197.126:33837 requested a full tablet report, sending...
I20260929 16:41:11.402590 841 ts_manager.cc:194] Registered new tserver with Master: a3a6dd4cf52b44e6a9d4c509d4b8a0bf (127.0.197.66:40843)
I20260929 16:41:11.403522 841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.66@KRBTEST.COM'} at 127.0.197.66:36399
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab.
Entry for principal kudu/127.0.197.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab.
I20260929 16:41:11.423594 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.67:0
--local_ip_for_outbound_sockets=127.0.197.67
--webserver_interface=127.0.197.67
--webserver_port=0
--tserver_master_addrs=127.0.197.126:33837
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab
--principal=kudu/127.0.197.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:35357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:41:11.791869 1170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:41:11.792270 1170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:41:11.792454 1170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:41:11.801692 1170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:41:11.801884 1170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:41:11.802084 1170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.67
I20260929 16:41:11.813704 1170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:35357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab
--principal=kudu/127.0.197.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.197.67
--webserver_port=0
--tserver_master_addrs=127.0.197.126:33837
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.67
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:41:11.815328 1170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:41:11.816752 1170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:41:11.831651 1176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:11.831733 1178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:11.831681 1175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:41:11.832715 1170 server_base.cc:1034] running on GCE node
I20260929 16:41:11.833402 1170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:41:11.834512 1170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:41:11.835798 1170 hybrid_clock.cc:648] HybridClock initialized: now 1790700071835748 us; error 46 us; skew 500 ppm
I20260929 16:41:11.836231 1170 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:41:11.836941 1030 heartbeater.cc:503] Master 127.0.197.126:33837 was elected leader, sending a full tablet report...
Sep 29 16:41:11 dist-test-slave-l31f krb5kdc[796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700071, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:41:11.840843 1170 init.cc:377] Logged in from keytab as kudu/127.0.197.67@KRBTEST.COM (short username kudu)
I20260929 16:41:11.842727 1170 webserver.cc:460] Webserver started at http://127.0.197.67:46653/ using document root <none> and password file <none>
I20260929 16:41:11.843629 1170 fs_manager.cc:359] Metadata directory not provided
I20260929 16:41:11.843755 1170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:41:11.844082 1170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:41:11.848292 1170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/ts-2/data/instance:
uuid: "3c03dbe3e46743599f8373d5fa2e2b68"
format_stamp: "Formatted at 2026-09-29 16:41:11 on dist-test-slave-l31f"
server_key: "3aa5f5d086bca39ba41d298a119b3510"
server_key_iv: "64ada042d3d2e77f72660316b113e2d4"
server_key_version: "encryptionkey@0"
I20260929 16:41:11.849252 1170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/ts-2/wal/instance:
uuid: "3c03dbe3e46743599f8373d5fa2e2b68"
format_stamp: "Formatted at 2026-09-29 16:41:11 on dist-test-slave-l31f"
server_key: "3aa5f5d086bca39ba41d298a119b3510"
server_key_iv: "64ada042d3d2e77f72660316b113e2d4"
server_key_version: "encryptionkey@0"
I20260929 16:41:11.855836 1170 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260929 16:41:11.861011 1185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:11.863191 1170 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260929 16:41:11.863426 1170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/ts-2/wal
uuid: "3c03dbe3e46743599f8373d5fa2e2b68"
format_stamp: "Formatted at 2026-09-29 16:41:11 on dist-test-slave-l31f"
server_key: "3aa5f5d086bca39ba41d298a119b3510"
server_key_iv: "64ada042d3d2e77f72660316b113e2d4"
server_key_version: "encryptionkey@0"
I20260929 16:41:11.863668 1170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:41:11.886057 1170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:41:11.891983 1170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:41:11.892354 1170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:41:11.893478 1170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:41:11.895364 1170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:41:11.895507 1170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:11.895637 1170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:41:11.895707 1170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:11.937511 1170 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.67:42915
I20260929 16:41:11.937729 1297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.67:42915 every 8 connection(s)
I20260929 16:41:11.939131 1170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
I20260929 16:41:11.944721 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 1170
I20260929 16:41:11.944940 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790700068255611-789-0/minicluster-data/ts-2/wal/instance
I20260929 16:41:11.945503 789 external_mini_cluster.cc:1428] Setting key 108fdffaac9689b18e3703a03bb11f3a
Sep 29 16:41:11 dist-test-slave-l31f krb5kdc[796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700071, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.67@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:41:11.964497 1300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:11.942462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:45145 (local address 127.0.197.126:33837)
0929 16:41:11.943079 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:11.943090 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:11.944018 (+ 928us) server_negotiation.cc:407] Connection header received
0929 16:41:11.946823 (+ 2805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:11.946828 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:11.946916 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:11.947119 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:11.949729 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:11.950820 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:11.952654 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:11.953090 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:11.953215 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:11.957388 (+ 4173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:11.957395 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:11.957399 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:11.957438 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:11.960510 (+ 3072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:11.961713 (+ 1203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:11.961719 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:11.961722 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:11.961892 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:11.962530 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:11.962536 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:11.962539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:11.962905 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:11.963141 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:11.964001 (+ 860us) server_negotiation.cc:299] Negotiation successful
0929 16:41:11.964218 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":304,"threads_started":1}
I20260929 16:41:11.965569 1301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:11.942892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:33837 (local address 127.0.197.67:45145)
0929 16:41:11.943786 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:11.943836 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:11.946437 (+ 2601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:11.947395 (+ 958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:11.947409 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:11.948764 (+ 1355us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:41:11.949529 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:11.949551 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:11.951091 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:11.951096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:11.952476 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:11.952489 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:11.953354 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:11.953359 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:11.953582 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:11.954274 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:41:11.954308 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:41:11.957099 (+ 2791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:41:11.960762 (+ 3663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:11.960774 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:11.960798 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:11.961533 (+ 735us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:11.962066 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:11.962071 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:11.962074 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:11.962359 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:11.963102 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:41:11.963114 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:41:11.963611 (+ 497us) client_negotiation.cc:769] Sending connection context
0929 16:41:11.963860 (+ 249us) client_negotiation.cc:240] Negotiation successful
0929 16:41:11.964337 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":395,"threads_started":1}
I20260929 16:41:11.967243 1298 heartbeater.cc:347] Connected to a master server at 127.0.197.126:33837
I20260929 16:41:11.967650 1298 heartbeater.cc:464] Registering TS with master...
I20260929 16:41:11.968745 1298 heartbeater.cc:511] Master 127.0.197.126:33837 requested a full tablet report, sending...
I20260929 16:41:11.970916 841 ts_manager.cc:194] Registered new tserver with Master: 3c03dbe3e46743599f8373d5fa2e2b68 (127.0.197.67:42915)
I20260929 16:41:11.971872 841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.67@KRBTEST.COM'} at 127.0.197.67:45145
I20260929 16:41:11.976279 789 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 16:41:11 dist-test-slave-l31f krb5kdc[796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700071, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 16:41:12 dist-test-slave-l31f krb5kdc[796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700071, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.197.65@KRBTEST.COM
I20260929 16:41:12.026003 1306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:12.007804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39930 (local address 127.0.197.65:44385)
0929 16:41:12.008477 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:12.008484 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:12.008513 (+ 29us) server_negotiation.cc:407] Connection header received
0929 16:41:12.008692 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:12.008699 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:12.008925 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:12.009077 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:12.010610 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:12.011792 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:12.013219 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:12.013850 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:12.013974 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:12.018743 (+ 4769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:12.018758 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:12.018775 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:12.018819 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:12.022000 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:12.023001 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:12.023100 (+ 99us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:12.023103 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:12.023247 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:12.024208 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:12.024212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:12.024214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:12.024603 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:12.024838 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:12.025278 (+ 440us) server_negotiation.cc:299] Negotiation successful
0929 16:41:12.025541 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":379,"threads_started":1}
W20260929 16:41:12.027403 945 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39930
Sep 29 16:41:12 dist-test-slave-l31f krb5kdc[796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 16:41:12 dist-test-slave-l31f krb5kdc[796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.65@KRBTEST.COM
I20260929 16:41:12.076264 1306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:12.059266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39934 (local address 127.0.197.65:44385)
0929 16:41:12.059447 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:12.059453 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:12.059994 (+ 541us) server_negotiation.cc:407] Connection header received
0929 16:41:12.060214 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:12.060219 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:12.060323 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:12.060421 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:12.062462 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:12.063305 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:12.064682 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:12.065121 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:12.065239 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:12.070494 (+ 5255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:12.070500 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:12.070504 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:12.070547 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:12.073352 (+ 2805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:12.074134 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:12.074139 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:12.074142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:12.074273 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:12.074714 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:12.074719 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:12.074721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:12.075102 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:12.075312 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:12.075679 (+ 367us) server_negotiation.cc:299] Negotiation successful
0929 16:41:12.075857 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 16:41:12.077998 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 902
I20260929 16:41:12.092093 1306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:12.079260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39942 (local address 127.0.197.65:44385)
0929 16:41:12.079452 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:12.079456 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:12.079725 (+ 269us) server_negotiation.cc:407] Connection header received
0929 16:41:12.079838 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:12.079842 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:12.079907 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:12.079995 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:12.081666 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:12.082462 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:12.083922 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:12.084352 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:12.084467 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:12.086938 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:12.086943 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:12.086946 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:12.086981 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:12.089545 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:12.090257 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:12.090261 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:12.090263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:12.090402 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:12.090732 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:12.090735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:12.090737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:12.091127 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:12.091350 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:12.091636 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 16:41:12.091808 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 16:41:12.092839 1025 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:41:12.177839 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 902
I20260929 16:41:12.195396 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 1036
Sep 29 16:41:12 dist-test-slave-l31f krb5kdc[796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.66@KRBTEST.COM
I20260929 16:41:12.213018 1313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:12.196354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43594 (local address 127.0.197.66:40843)
0929 16:41:12.196911 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:12.196918 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:12.196943 (+ 25us) server_negotiation.cc:407] Connection header received
0929 16:41:12.197002 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:12.197008 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:12.197220 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:12.197407 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:12.198638 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:12.199868 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:12.201418 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:12.202152 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:12.202293 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:12.206923 (+ 4630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:12.206939 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:12.206957 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:12.207110 (+ 153us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:12.210123 (+ 3013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:12.210932 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:12.210940 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:12.210942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:12.211187 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:12.211588 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:12.211592 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:12.211594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:12.211962 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:12.212201 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:12.212499 (+ 298us) server_negotiation.cc:299] Negotiation successful
0929 16:41:12.212709 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":290,"threads_started":1}
I20260929 16:41:12.214375 1159 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:41:12.300472 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 1036
I20260929 16:41:12.323741 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 1170
Sep 29 16:41:12 dist-test-slave-l31f krb5kdc[796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.67@KRBTEST.COM
I20260929 16:41:12.342414 1315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:12.325066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40262 (local address 127.0.197.67:42915)
0929 16:41:12.325732 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:12.325739 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:12.325769 (+ 30us) server_negotiation.cc:407] Connection header received
0929 16:41:12.325833 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:12.325840 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:12.326053 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:12.326205 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:12.327631 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:12.328736 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:12.330111 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:12.330732 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:12.330850 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:12.335593 (+ 4743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:12.335609 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:12.335625 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:12.335670 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:12.338972 (+ 3302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:12.340068 (+ 1096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:12.340076 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:12.340085 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:12.340229 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:12.340741 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:12.340745 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:12.340747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:12.341143 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:12.341403 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:12.341881 (+ 478us) server_negotiation.cc:299] Negotiation successful
0929 16:41:12.342110 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":378,"threads_started":1}
I20260929 16:41:12.343954 1293 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:41:12.429021 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 1170
I20260929 16:41:12.445861 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 809
I20260929 16:41:12.447178 871 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:41:12.551365 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 809
2026-09-29T16:41:12Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4252 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:41:12 dist-test-slave-l31f krb5kdc[1320](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:41:12 dist-test-slave-l31f krb5kdc[1320](info): set up 2 sockets
Sep 29 16:41:12 dist-test-slave-l31f krb5kdc[1320](info): commencing operation
krb5kdc: starting...
W20260929 16:41:13.724522 789 mini_kdc.cc:121] Time spent starting KDC: real 1.127s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260929 16:41:14.051604 789 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.327s user 0.000s sys 0.006s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260929 16:41:14.278436 789 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.227s user 0.000s sys 0.004s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260929 16:41:14.507251 789 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.229s user 0.000s sys 0.004s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/krb5kdc/test-user.keytab.
W20260929 16:41:15.290167 789 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.783s user 0.000s sys 0.005s
Sep 29 16:41:15 dist-test-slave-l31f krb5kdc[1320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:41:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:41:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.197.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
W20260929 16:41:16.304368 789 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.197.126: real 0.961s user 0.004s sys 0.005s
I20260929 16:41:16.305819 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:44117
--webserver_interface=127.0.197.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:37981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.126:44117
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:41:16.699671 1333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:41:16.700022 1333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:41:16.700187 1333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:41:16.708736 1333 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:41:16.708899 1333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:41:16.709002 1333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:41:16.709091 1333 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:41:16.709183 1333 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:41:16.719969 1333 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:37981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.197.126:44117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:44117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.197.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:41:16.721510 1333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:41:16.723055 1333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:41:16.736864 1338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:16.736846 1341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:16.737079 1339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:41:16.737663 1333 server_base.cc:1034] running on GCE node
I20260929 16:41:16.738745 1333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:41:16.741540 1333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:41:16.743145 1333 hybrid_clock.cc:648] HybridClock initialized: now 1790700076743072 us; error 59 us; skew 500 ppm
I20260929 16:41:16.743626 1333 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:41:16 dist-test-slave-l31f krb5kdc[1320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:41:16.747385 1333 init.cc:377] Logged in from keytab as kudu/127.0.197.126@KRBTEST.COM (short username kudu)
I20260929 16:41:16.749388 1333 webserver.cc:460] Webserver started at http://127.0.197.126:36033/ using document root <none> and password file <none>
I20260929 16:41:16.750224 1333 fs_manager.cc:359] Metadata directory not provided
I20260929 16:41:16.750375 1333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:41:16.750736 1333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:41:16.754172 1333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/master-0/data/instance:
uuid: "88849a405a6e49958424308ed8fc11a7"
format_stamp: "Formatted at 2026-09-29 16:41:16 on dist-test-slave-l31f"
server_key: "657e3e80ff4d17a1adc50a305577b201"
server_key_iv: "3ba93b39831a7d4b0bb1bab8f034836f"
server_key_version: "encryptionkey@0"
I20260929 16:41:16.755195 1333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/master-0/wal/instance:
uuid: "88849a405a6e49958424308ed8fc11a7"
format_stamp: "Formatted at 2026-09-29 16:41:16 on dist-test-slave-l31f"
server_key: "657e3e80ff4d17a1adc50a305577b201"
server_key_iv: "3ba93b39831a7d4b0bb1bab8f034836f"
server_key_version: "encryptionkey@0"
I20260929 16:41:16.761312 1333 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 16:41:16.765600 1348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:16.767329 1333 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:41:16.767585 1333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "88849a405a6e49958424308ed8fc11a7"
format_stamp: "Formatted at 2026-09-29 16:41:16 on dist-test-slave-l31f"
server_key: "657e3e80ff4d17a1adc50a305577b201"
server_key_iv: "3ba93b39831a7d4b0bb1bab8f034836f"
server_key_version: "encryptionkey@0"
I20260929 16:41:16.767827 1333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:41:16.789708 1333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:41:16.794660 1333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:41:16.795079 1333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:41:16.818917 1333 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.126:44117
I20260929 16:41:16.818935 1399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.126:44117 every 8 connection(s)
I20260929 16:41:16.820618 1333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:41:16.825448 1400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:41:16.827638 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 1333
I20260929 16:41:16.827867 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/master-0/wal/instance
I20260929 16:41:16.829239 789 external_mini_cluster.cc:1428] Setting key 4f5414aad5673d8b87ef201a7f5d982b
I20260929 16:41:16.841111 1400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88849a405a6e49958424308ed8fc11a7: Bootstrap starting.
I20260929 16:41:16.845886 1400 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88849a405a6e49958424308ed8fc11a7: Neither blocks nor log segments found. Creating new log.
Sep 29 16:41:16 dist-test-slave-l31f krb5kdc[1320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:41:16.847378 1400 log.cc:824] T 00000000000000000000000000000000 P 88849a405a6e49958424308ed8fc11a7: Log is configured to *not* fsync() on all Append() calls
I20260929 16:41:16.851598 1400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88849a405a6e49958424308ed8fc11a7: No bootstrap required, opened a new log
I20260929 16:41:16.858978 1403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:16.832028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40830 (local address 127.0.197.126:44117)
0929 16:41:16.833078 (+ 1050us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:16.833096 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:16.833158 (+ 62us) server_negotiation.cc:407] Connection header received
0929 16:41:16.835299 (+ 2141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:16.835330 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:16.835856 (+ 526us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:16.836305 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:16.838396 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:16.840117 (+ 1721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:16.842078 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:16.842878 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:16.843162 (+ 284us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:16.848858 (+ 5696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:16.848875 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:16.848909 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:16.848969 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:16.853224 (+ 4255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:16.854322 (+ 1098us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:16.854330 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:16.854337 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:16.854565 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:16.855126 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:16.855134 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:16.855140 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:16.856151 (+ 1011us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:16.856496 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:16.857116 (+ 620us) server_negotiation.cc:299] Negotiation successful
0929 16:41:16.857714 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":781,"thread_start_us":502,"threads_started":1}
I20260929 16:41:16.861641 1400 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88849a405a6e49958424308ed8fc11a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88849a405a6e49958424308ed8fc11a7" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 44117 } }
I20260929 16:41:16.862583 1400 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88849a405a6e49958424308ed8fc11a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:41:16.862737 1400 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88849a405a6e49958424308ed8fc11a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88849a405a6e49958424308ed8fc11a7, State: Initialized, Role: FOLLOWER
I20260929 16:41:16.863528 1400 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88849a405a6e49958424308ed8fc11a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88849a405a6e49958424308ed8fc11a7" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 44117 } }
I20260929 16:41:16.864108 1400 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88849a405a6e49958424308ed8fc11a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:41:16.864464 1400 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88849a405a6e49958424308ed8fc11a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:41:16.864625 1400 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88849a405a6e49958424308ed8fc11a7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:41:16.901666 1400 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88849a405a6e49958424308ed8fc11a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88849a405a6e49958424308ed8fc11a7" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 44117 } }
I20260929 16:41:16.902397 1400 leader_election.cc:302] T 00000000000000000000000000000000 P 88849a405a6e49958424308ed8fc11a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88849a405a6e49958424308ed8fc11a7; no voters:
I20260929 16:41:16.903556 1400 leader_election.cc:288] T 00000000000000000000000000000000 P 88849a405a6e49958424308ed8fc11a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:41:16.903582 1405 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88849a405a6e49958424308ed8fc11a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:41:16.904706 1405 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88849a405a6e49958424308ed8fc11a7 [term 1 LEADER]: Becoming Leader. State: Replica: 88849a405a6e49958424308ed8fc11a7, State: Running, Role: LEADER
I20260929 16:41:16.905546 1405 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88849a405a6e49958424308ed8fc11a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88849a405a6e49958424308ed8fc11a7" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 44117 } }
I20260929 16:41:16.908854 1400 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88849a405a6e49958424308ed8fc11a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:41:16.909974 1407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88849a405a6e49958424308ed8fc11a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88849a405a6e49958424308ed8fc11a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88849a405a6e49958424308ed8fc11a7" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 44117 } } }
I20260929 16:41:16.909920 1406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88849a405a6e49958424308ed8fc11a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88849a405a6e49958424308ed8fc11a7. Latest consensus state: current_term: 1 leader_uuid: "88849a405a6e49958424308ed8fc11a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88849a405a6e49958424308ed8fc11a7" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 44117 } } }
I20260929 16:41:16.910519 1406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88849a405a6e49958424308ed8fc11a7 [sys.catalog]: This master's current role is: LEADER
I20260929 16:41:16.910897 1407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88849a405a6e49958424308ed8fc11a7 [sys.catalog]: This master's current role is: LEADER
I20260929 16:41:16.912505 1410 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:41:16.919045 1410 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:41:16.930447 1410 catalog_manager.cc:1347] Generated new cluster ID: 0bd2dd347ae2472698d12b50bd92aad0
I20260929 16:41:16.930671 1410 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:41:16.948329 1410 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:41:16.949856 1410 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:41:16.958688 1410 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88849a405a6e49958424308ed8fc11a7: Generated new TSK 0
I20260929 16:41:16.959972 1410 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.197.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab.
Entry for principal kudu/127.0.197.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab.
W20260929 16:41:17.784873 789 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.197.65: real 0.810s user 0.005s sys 0.005s
I20260929 16:41:17.786190 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.65:0
--local_ip_for_outbound_sockets=127.0.197.65
--webserver_interface=127.0.197.65
--webserver_port=0
--tserver_master_addrs=127.0.197.126:44117
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab
--principal=kudu/127.0.197.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:37981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:41:18.146634 1426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:41:18.147603 1426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:41:18.147868 1426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:41:18.155880 1426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:41:18.156023 1426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:41:18.156211 1426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.65
I20260929 16:41:18.167547 1426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:37981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab
--principal=kudu/127.0.197.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.197.65
--webserver_port=0
--tserver_master_addrs=127.0.197.126:44117
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.65
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:41:18.168998 1426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:41:18.170460 1426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:41:18.186858 1434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:18.186817 1431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:18.186937 1432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:41:18.188197 1426 server_base.cc:1034] running on GCE node
I20260929 16:41:18.189177 1426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:41:18.190846 1426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:41:18.192356 1426 hybrid_clock.cc:648] HybridClock initialized: now 1790700078192270 us; error 159 us; skew 500 ppm
I20260929 16:41:18.192828 1426 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:41:18 dist-test-slave-l31f krb5kdc[1320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:41:18.197410 1426 init.cc:377] Logged in from keytab as kudu/127.0.197.65@KRBTEST.COM (short username kudu)
I20260929 16:41:18.199577 1426 webserver.cc:460] Webserver started at http://127.0.197.65:43549/ using document root <none> and password file <none>
I20260929 16:41:18.200495 1426 fs_manager.cc:359] Metadata directory not provided
I20260929 16:41:18.200641 1426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:41:18.201009 1426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:41:18.204362 1426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/ts-0/data/instance:
uuid: "969389c1869148b7ba94d01f14de514d"
format_stamp: "Formatted at 2026-09-29 16:41:18 on dist-test-slave-l31f"
server_key: "6b033a6e988dea0bc4793bf134aab696"
server_key_iv: "2ccc0ba63f184cb6d08ddd3b461fd727"
server_key_version: "encryptionkey@0"
I20260929 16:41:18.205309 1426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/ts-0/wal/instance:
uuid: "969389c1869148b7ba94d01f14de514d"
format_stamp: "Formatted at 2026-09-29 16:41:18 on dist-test-slave-l31f"
server_key: "6b033a6e988dea0bc4793bf134aab696"
server_key_iv: "2ccc0ba63f184cb6d08ddd3b461fd727"
server_key_version: "encryptionkey@0"
I20260929 16:41:18.211449 1426 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 16:41:18.215727 1441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:18.217417 1426 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 16:41:18.217646 1426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/ts-0/wal
uuid: "969389c1869148b7ba94d01f14de514d"
format_stamp: "Formatted at 2026-09-29 16:41:18 on dist-test-slave-l31f"
server_key: "6b033a6e988dea0bc4793bf134aab696"
server_key_iv: "2ccc0ba63f184cb6d08ddd3b461fd727"
server_key_version: "encryptionkey@0"
I20260929 16:41:18.217881 1426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:41:18.244040 1426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:41:18.249578 1426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:41:18.249924 1426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:41:18.251117 1426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:41:18.252630 1426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:41:18.252714 1426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:18.252859 1426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:41:18.252931 1426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:18.292058 1426 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.65:44753
I20260929 16:41:18.292176 1553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.65:44753 every 8 connection(s)
I20260929 16:41:18.293465 1426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
I20260929 16:41:18.297441 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 1426
I20260929 16:41:18.297704 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/ts-0/wal/instance
I20260929 16:41:18.298516 789 external_mini_cluster.cc:1428] Setting key 41291044b2a7c021ee5311db1e809cbc
Sep 29 16:41:18 dist-test-slave-l31f krb5kdc[1320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.65@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.197.66@KRBTEST.COM; defaulting to no policy
I20260929 16:41:18.320458 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:18.296544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:34929 (local address 127.0.197.126:44117)
0929 16:41:18.297188 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:18.297197 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:18.298508 (+ 1311us) server_negotiation.cc:407] Connection header received
0929 16:41:18.301100 (+ 2592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:18.301105 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:18.301183 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:18.301282 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:18.303199 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:18.304492 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:18.306722 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:18.307227 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:18.307340 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:18.312538 (+ 5198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:18.312544 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:18.312548 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:18.312587 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:18.316710 (+ 4123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:18.317735 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:18.317744 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:18.317749 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:18.317925 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:18.318443 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:18.318448 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:18.318453 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:18.318794 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:18.318912 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:18.319933 (+ 1021us) server_negotiation.cc:299] Negotiation successful
0929 16:41:18.320157 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":281,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.66@KRBTEST.COM" created.
I20260929 16:41:18.321695 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:18.297260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:44117 (local address 127.0.197.65:34929)
0929 16:41:18.298273 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:18.298332 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:18.300695 (+ 2363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:18.301592 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:18.301602 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:18.302196 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:41:18.302841 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:18.302860 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:18.304720 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:18.304725 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:18.306486 (+ 1761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:18.306510 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:18.307495 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:18.307505 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:18.307893 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:18.308783 (+ 890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:41:18.308821 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:41:18.312213 (+ 3392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:41:18.316985 (+ 4772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:18.316995 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:18.317013 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:18.317564 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:18.318061 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:18.318065 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:18.318067 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:18.318271 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:18.318991 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:41:18.319000 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:41:18.319510 (+ 510us) client_negotiation.cc:769] Sending connection context
0929 16:41:18.319825 (+ 315us) client_negotiation.cc:240] Negotiation successful
0929 16:41:18.320266 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":357,"threads_started":1}
I20260929 16:41:18.323980 1554 heartbeater.cc:347] Connected to a master server at 127.0.197.126:44117
I20260929 16:41:18.324345 1554 heartbeater.cc:464] Registering TS with master...
I20260929 16:41:18.325240 1554 heartbeater.cc:511] Master 127.0.197.126:44117 requested a full tablet report, sending...
I20260929 16:41:18.327929 1365 ts_manager.cc:194] Registered new tserver with Master: 969389c1869148b7ba94d01f14de514d (127.0.197.65:44753)
I20260929 16:41:18.329449 1365 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.65@KRBTEST.COM'} at 127.0.197.65:34929
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab.
Entry for principal kudu/127.0.197.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab.
I20260929 16:41:18.351903 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.66:0
--local_ip_for_outbound_sockets=127.0.197.66
--webserver_interface=127.0.197.66
--webserver_port=0
--tserver_master_addrs=127.0.197.126:44117
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab
--principal=kudu/127.0.197.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:37981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:41:18.702884 1560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:41:18.703326 1560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:41:18.703518 1560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:41:18.711483 1560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:41:18.711692 1560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:41:18.711895 1560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.66
I20260929 16:41:18.723155 1560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:37981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab
--principal=kudu/127.0.197.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.197.66
--webserver_port=0
--tserver_master_addrs=127.0.197.126:44117
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.66
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:41:18.724649 1560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:41:18.726043 1560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:41:18.740928 1565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:18.740969 1566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:18.741052 1568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:41:18.742173 1560 server_base.cc:1034] running on GCE node
I20260929 16:41:18.742959 1560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:41:18.744233 1560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:41:18.745486 1560 hybrid_clock.cc:648] HybridClock initialized: now 1790700078745434 us; error 45 us; skew 500 ppm
I20260929 16:41:18.745900 1560 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:41:18 dist-test-slave-l31f krb5kdc[1320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:41:18.749445 1560 init.cc:377] Logged in from keytab as kudu/127.0.197.66@KRBTEST.COM (short username kudu)
I20260929 16:41:18.751390 1560 webserver.cc:460] Webserver started at http://127.0.197.66:42867/ using document root <none> and password file <none>
I20260929 16:41:18.752241 1560 fs_manager.cc:359] Metadata directory not provided
I20260929 16:41:18.752360 1560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:41:18.752781 1560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:41:18.756606 1560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/ts-1/data/instance:
uuid: "659d0a10b5354643b948f00aa0297daf"
format_stamp: "Formatted at 2026-09-29 16:41:18 on dist-test-slave-l31f"
server_key: "edaa372fdd41c859f6b40d537138de15"
server_key_iv: "8e431a0b7cc98adb7fa3f19411017e13"
server_key_version: "encryptionkey@0"
I20260929 16:41:18.757530 1560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/ts-1/wal/instance:
uuid: "659d0a10b5354643b948f00aa0297daf"
format_stamp: "Formatted at 2026-09-29 16:41:18 on dist-test-slave-l31f"
server_key: "edaa372fdd41c859f6b40d537138de15"
server_key_iv: "8e431a0b7cc98adb7fa3f19411017e13"
server_key_version: "encryptionkey@0"
I20260929 16:41:18.763932 1560 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 16:41:18.768170 1575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:18.769834 1560 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:41:18.770033 1560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/ts-1/wal
uuid: "659d0a10b5354643b948f00aa0297daf"
format_stamp: "Formatted at 2026-09-29 16:41:18 on dist-test-slave-l31f"
server_key: "edaa372fdd41c859f6b40d537138de15"
server_key_iv: "8e431a0b7cc98adb7fa3f19411017e13"
server_key_version: "encryptionkey@0"
I20260929 16:41:18.770319 1560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:41:18.789510 1560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:41:18.794639 1560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:41:18.794975 1560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:41:18.796085 1560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:41:18.797556 1560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:41:18.797631 1560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:18.797770 1560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:41:18.797822 1560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:18.839149 1560 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.66:38981
I20260929 16:41:18.839277 1687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.66:38981 every 8 connection(s)
I20260929 16:41:18.840631 1560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
I20260929 16:41:18.842085 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 1560
I20260929 16:41:18.842303 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/ts-1/wal/instance
I20260929 16:41:18.842864 789 external_mini_cluster.cc:1428] Setting key c7801d05f76be273dc9e27795b12f43f
Sep 29 16:41:18 dist-test-slave-l31f krb5kdc[1320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.66@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.197.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.67@KRBTEST.COM" created.
I20260929 16:41:18.866330 1690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:18.844030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:37625 (local address 127.0.197.126:44117)
0929 16:41:18.844530 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:18.844535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:18.846020 (+ 1485us) server_negotiation.cc:407] Connection header received
0929 16:41:18.848111 (+ 2091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:18.848120 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:18.848242 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:18.848378 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:18.850289 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:18.851146 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:18.853288 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:18.853756 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:18.853913 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:18.859132 (+ 5219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:18.859144 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:18.859152 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:18.859215 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:18.862330 (+ 3115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:18.863614 (+ 1284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:18.863621 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:18.863625 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:18.863791 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:18.864344 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:18.864349 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:18.864354 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:18.864648 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:18.864819 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:18.865645 (+ 826us) server_negotiation.cc:299] Negotiation successful
0929 16:41:18.865911 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":274,"threads_started":1}
I20260929 16:41:18.867559 1691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:18.844897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:44117 (local address 127.0.197.66:37625)
0929 16:41:18.845813 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:18.845875 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:18.847763 (+ 1888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:18.848686 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:18.848696 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:18.849286 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:41:18.849970 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:18.849991 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:18.851346 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:18.851350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:18.853066 (+ 1716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:18.853084 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:18.854032 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:18.854042 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:18.854332 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:18.855222 (+ 890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:41:18.855251 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:41:18.858590 (+ 3339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:41:18.862591 (+ 4001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:18.862602 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:18.862621 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:18.863430 (+ 809us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:18.863953 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:18.863959 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:18.863964 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:18.864199 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:18.864795 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:41:18.864804 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:41:18.865219 (+ 415us) client_negotiation.cc:769] Sending connection context
0929 16:41:18.865532 (+ 313us) client_negotiation.cc:240] Negotiation successful
0929 16:41:18.866156 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":389,"threads_started":1}
I20260929 16:41:18.869573 1688 heartbeater.cc:347] Connected to a master server at 127.0.197.126:44117
I20260929 16:41:18.869936 1688 heartbeater.cc:464] Registering TS with master...
I20260929 16:41:18.870774 1688 heartbeater.cc:511] Master 127.0.197.126:44117 requested a full tablet report, sending...
I20260929 16:41:18.872766 1365 ts_manager.cc:194] Registered new tserver with Master: 659d0a10b5354643b948f00aa0297daf (127.0.197.66:38981)
I20260929 16:41:18.873636 1365 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.66@KRBTEST.COM'} at 127.0.197.66:37625
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab.
Entry for principal kudu/127.0.197.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab.
I20260929 16:41:18.892230 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.67:0
--local_ip_for_outbound_sockets=127.0.197.67
--webserver_interface=127.0.197.67
--webserver_port=0
--tserver_master_addrs=127.0.197.126:44117
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab
--principal=kudu/127.0.197.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:37981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:41:19.256377 1694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:41:19.256713 1694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:41:19.256814 1694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:41:19.264457 1694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:41:19.264600 1694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:41:19.264721 1694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.67
I20260929 16:41:19.275421 1694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:37981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab
--principal=kudu/127.0.197.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.197.67
--webserver_port=0
--tserver_master_addrs=127.0.197.126:44117
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.67
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:41:19.276824 1694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:41:19.278182 1694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:41:19.293767 1700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:19.294543 1702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:19.294786 1699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:41:19.295140 1694 server_base.cc:1034] running on GCE node
I20260929 16:41:19.295820 1694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:41:19.296885 1694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:41:19.298156 1694 hybrid_clock.cc:648] HybridClock initialized: now 1790700079298097 us; error 52 us; skew 500 ppm
I20260929 16:41:19.298542 1694 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:41:19 dist-test-slave-l31f krb5kdc[1320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700079, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:41:19.302172 1694 init.cc:377] Logged in from keytab as kudu/127.0.197.67@KRBTEST.COM (short username kudu)
I20260929 16:41:19.304188 1694 webserver.cc:460] Webserver started at http://127.0.197.67:43815/ using document root <none> and password file <none>
I20260929 16:41:19.304939 1694 fs_manager.cc:359] Metadata directory not provided
I20260929 16:41:19.305063 1694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:41:19.305338 1694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:41:19.308611 1694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/ts-2/data/instance:
uuid: "7d549fb650c845e39af8c1ccf453b3b1"
format_stamp: "Formatted at 2026-09-29 16:41:19 on dist-test-slave-l31f"
server_key: "84edd07347bdefef072202ce234ecd16"
server_key_iv: "329d9f7f10b05f1751f992024ef144ef"
server_key_version: "encryptionkey@0"
I20260929 16:41:19.309613 1694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/ts-2/wal/instance:
uuid: "7d549fb650c845e39af8c1ccf453b3b1"
format_stamp: "Formatted at 2026-09-29 16:41:19 on dist-test-slave-l31f"
server_key: "84edd07347bdefef072202ce234ecd16"
server_key_iv: "329d9f7f10b05f1751f992024ef144ef"
server_key_version: "encryptionkey@0"
I20260929 16:41:19.316390 1694 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260929 16:41:19.321173 1709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:19.323145 1694 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20260929 16:41:19.323333 1694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/ts-2/wal
uuid: "7d549fb650c845e39af8c1ccf453b3b1"
format_stamp: "Formatted at 2026-09-29 16:41:19 on dist-test-slave-l31f"
server_key: "84edd07347bdefef072202ce234ecd16"
server_key_iv: "329d9f7f10b05f1751f992024ef144ef"
server_key_version: "encryptionkey@0"
I20260929 16:41:19.323552 1694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:41:19.333559 1554 heartbeater.cc:503] Master 127.0.197.126:44117 was elected leader, sending a full tablet report...
I20260929 16:41:19.339468 1694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:41:19.344362 1694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:41:19.344697 1694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:41:19.345775 1694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:41:19.347355 1694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:41:19.347441 1694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:19.347579 1694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:41:19.347630 1694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:19.383104 1694 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.67:35149
I20260929 16:41:19.383157 1821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.67:35149 every 8 connection(s)
I20260929 16:41:19.384621 1694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
I20260929 16:41:19.392061 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 1694
I20260929 16:41:19.392263 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790700068255611-789-0/minicluster-data/ts-2/wal/instance
I20260929 16:41:19.392777 789 external_mini_cluster.cc:1428] Setting key aec7fa596d97c5c52d0828e40964e73c
Sep 29 16:41:19 dist-test-slave-l31f krb5kdc[1320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700079, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.67@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:41:19.412930 1824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:19.387945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:40301 (local address 127.0.197.126:44117)
0929 16:41:19.390183 (+ 2238us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:19.390192 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:19.390539 (+ 347us) server_negotiation.cc:407] Connection header received
0929 16:41:19.393653 (+ 3114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:19.393662 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:19.393779 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:19.393978 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:19.397043 (+ 3065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:19.398003 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:19.400192 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:19.400795 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:19.400953 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:19.405201 (+ 4248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:19.405206 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:19.405210 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:19.405247 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:19.408551 (+ 3304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:19.409532 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:19.409539 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:19.409543 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:19.409730 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:19.410224 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:19.410228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:19.410230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:19.410703 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:19.410834 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:19.412426 (+ 1592us) server_negotiation.cc:299] Negotiation successful
0929 16:41:19.412627 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2092,"thread_start_us":1964,"threads_started":1}
I20260929 16:41:19.414031 1825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:19.388778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:44117 (local address 127.0.197.67:40301)
0929 16:41:19.390348 (+ 1570us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:19.390396 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:19.392899 (+ 2503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:19.395166 (+ 2267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:19.395185 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:19.396025 (+ 840us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:41:19.396839 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:19.396867 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:19.398197 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:19.398202 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:19.399968 (+ 1766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:19.399986 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:19.401117 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:19.401126 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:19.401375 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:19.402148 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:41:19.402190 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:41:19.404892 (+ 2702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:41:19.408729 (+ 3837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:19.408741 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:19.408763 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:19.409313 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:19.409861 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:19.409864 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:19.409867 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:19.410066 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:19.410931 (+ 865us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:41:19.410941 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:41:19.411780 (+ 839us) client_negotiation.cc:769] Sending connection context
0929 16:41:19.412167 (+ 387us) client_negotiation.cc:240] Negotiation successful
0929 16:41:19.412653 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":857,"thread_start_us":476,"threads_started":1}
I20260929 16:41:19.416081 1822 heartbeater.cc:347] Connected to a master server at 127.0.197.126:44117
I20260929 16:41:19.416457 1822 heartbeater.cc:464] Registering TS with master...
I20260929 16:41:19.417287 1822 heartbeater.cc:511] Master 127.0.197.126:44117 requested a full tablet report, sending...
I20260929 16:41:19.419329 1365 ts_manager.cc:194] Registered new tserver with Master: 7d549fb650c845e39af8c1ccf453b3b1 (127.0.197.67:35149)
I20260929 16:41:19.420379 1365 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.67@KRBTEST.COM'} at 127.0.197.67:40301
I20260929 16:41:19.422686 789 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:41:19.444065 1824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:19.430230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40838 (local address 127.0.197.126:44117)
0929 16:41:19.430420 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:19.430431 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:19.431125 (+ 694us) server_negotiation.cc:407] Connection header received
0929 16:41:19.431284 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:19.431288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:19.431361 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:19.431460 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:19.433042 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:19.433815 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:19.435726 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:19.436165 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:19.436286 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:19.438755 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:19.438760 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:19.438763 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:19.438800 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:19.441330 (+ 2530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:19.442083 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:19.442088 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:19.442090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:19.442229 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:19.442571 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:19.442574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:19.442576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:19.442917 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:19.443203 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:19.443609 (+ 406us) server_negotiation.cc:299] Negotiation successful
0929 16:41:19.443810 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 16:41:19.451458 1364 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40838:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:41:19.458940 1364 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:41:19.488061 1835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:19.473803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.67:35149 (local address 127.0.0.1:55776)
0929 16:41:19.477648 (+ 3845us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:19.477726 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:19.477962 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:19.479171 (+ 1209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:19.479181 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:19.479213 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:41:19.479699 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:19.479718 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:19.481813 (+ 2095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:19.481819 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:19.484322 (+ 2503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:19.484337 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:19.486880 (+ 2543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:19.486886 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:19.487142 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:19.487203 (+ 61us) client_negotiation.cc:769] Sending connection context
0929 16:41:19.487347 (+ 144us) client_negotiation.cc:240] Negotiation successful
0929 16:41:19.487628 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3529,"thread_start_us":3383,"threads_started":1}
I20260929 16:41:19.489256 1836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:19.474154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55776 (local address 127.0.197.67:35149)
0929 16:41:19.475800 (+ 1646us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:19.475814 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:19.478201 (+ 2387us) server_negotiation.cc:407] Connection header received
0929 16:41:19.478312 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:19.478320 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:19.478610 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:19.478808 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:41:19.479939 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:19.481639 (+ 1700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:19.484534 (+ 2895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:19.486227 (+ 1693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:19.486383 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:19.486710 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:19.488660 (+ 1950us) server_negotiation.cc:299] Negotiation successful
0929 16:41:19.488877 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1288,"thread_start_us":1164,"threads_started":1}
I20260929 16:41:19.492592 1838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:19.475151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59960 (local address 127.0.197.65:44753)
0929 16:41:19.476651 (+ 1500us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:19.476660 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:19.477856 (+ 1196us) server_negotiation.cc:407] Connection header received
0929 16:41:19.479672 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:19.479683 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:19.479959 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:19.480156 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:41:19.481094 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:19.482701 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:19.486780 (+ 4079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:19.488544 (+ 1764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:19.488678 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:19.488921 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:19.491492 (+ 2571us) server_negotiation.cc:299] Negotiation successful
0929 16:41:19.492205 (+ 713us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1307,"thread_start_us":1159,"threads_started":1}
I20260929 16:41:19.493499 1837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:19.474394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:44753 (local address 127.0.0.1:59960)
0929 16:41:19.477648 (+ 3254us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:19.477726 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:19.477962 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:19.480455 (+ 2493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:19.480459 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:19.480487 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:41:19.480874 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:19.480886 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:19.484385 (+ 3499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:19.484392 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:19.486546 (+ 2154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:19.486562 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:19.490958 (+ 4396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:19.490966 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:19.491212 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:19.491250 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 16:41:19.492836 (+ 1586us) client_negotiation.cc:240] Negotiation successful
0929 16:41:19.493045 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3025,"thread_start_us":2854,"threads_started":1}
I20260929 16:41:19.496038 1757 tablet_service.cc:1467] Processing CreateTablet for tablet a8102f42e7874aec9b3c55678a691cc2 (DEFAULT_TABLE table=test-table [id=ae94e19a6e024d76b24bf2de5d882da1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:41:19.497726 1757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8102f42e7874aec9b3c55678a691cc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:41:19.498958 1489 tablet_service.cc:1467] Processing CreateTablet for tablet a8102f42e7874aec9b3c55678a691cc2 (DEFAULT_TABLE table=test-table [id=ae94e19a6e024d76b24bf2de5d882da1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:41:19.500453 1489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8102f42e7874aec9b3c55678a691cc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:41:19.504832 1835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:19.478451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.66:38981 (local address 127.0.0.1:50256)
0929 16:41:19.490191 (+ 11740us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:19.490216 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:19.490343 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:19.496257 (+ 5914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:19.496262 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:19.496294 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:41:19.496508 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:19.496521 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:19.498700 (+ 2179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:19.498708 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:19.500867 (+ 2159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:19.500889 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:19.503844 (+ 2955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:19.503865 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:19.504103 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:19.504150 (+ 47us) client_negotiation.cc:769] Sending connection context
0929 16:41:19.504277 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 16:41:19.504473 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11589,"thread_start_us":15784,"threads_started":1}
I20260929 16:41:19.504913 1840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:19.478681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50256 (local address 127.0.197.66:38981)
0929 16:41:19.495252 (+ 16571us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:19.495263 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:19.495299 (+ 36us) server_negotiation.cc:407] Connection header received
0929 16:41:19.495436 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:19.495445 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:19.495745 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:19.495980 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:41:19.496751 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:19.498493 (+ 1742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:19.501131 (+ 2638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:19.503618 (+ 2487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:19.503803 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:19.504070 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:19.504349 (+ 279us) server_negotiation.cc:299] Negotiation successful
0929 16:41:19.504547 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16396,"thread_start_us":15868,"threads_started":1}
I20260929 16:41:19.510514 1623 tablet_service.cc:1467] Processing CreateTablet for tablet a8102f42e7874aec9b3c55678a691cc2 (DEFAULT_TABLE table=test-table [id=ae94e19a6e024d76b24bf2de5d882da1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:41:19.512856 1623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8102f42e7874aec9b3c55678a691cc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:41:19.514072 1841 tablet_bootstrap.cc:492] T a8102f42e7874aec9b3c55678a691cc2 P 7d549fb650c845e39af8c1ccf453b3b1: Bootstrap starting.
I20260929 16:41:19.517714 1841 tablet_bootstrap.cc:654] T a8102f42e7874aec9b3c55678a691cc2 P 7d549fb650c845e39af8c1ccf453b3b1: Neither blocks nor log segments found. Creating new log.
I20260929 16:41:19.518752 1842 tablet_bootstrap.cc:492] T a8102f42e7874aec9b3c55678a691cc2 P 969389c1869148b7ba94d01f14de514d: Bootstrap starting.
I20260929 16:41:19.519381 1841 log.cc:824] T a8102f42e7874aec9b3c55678a691cc2 P 7d549fb650c845e39af8c1ccf453b3b1: Log is configured to *not* fsync() on all Append() calls
I20260929 16:41:19.523401 1842 tablet_bootstrap.cc:654] T a8102f42e7874aec9b3c55678a691cc2 P 969389c1869148b7ba94d01f14de514d: Neither blocks nor log segments found. Creating new log.
I20260929 16:41:19.524959 1842 log.cc:824] T a8102f42e7874aec9b3c55678a691cc2 P 969389c1869148b7ba94d01f14de514d: Log is configured to *not* fsync() on all Append() calls
I20260929 16:41:19.525086 1841 tablet_bootstrap.cc:492] T a8102f42e7874aec9b3c55678a691cc2 P 7d549fb650c845e39af8c1ccf453b3b1: No bootstrap required, opened a new log
I20260929 16:41:19.525897 1841 ts_tablet_manager.cc:1397] T a8102f42e7874aec9b3c55678a691cc2 P 7d549fb650c845e39af8c1ccf453b3b1: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260929 16:41:19.530925 1845 tablet_bootstrap.cc:492] T a8102f42e7874aec9b3c55678a691cc2 P 659d0a10b5354643b948f00aa0297daf: Bootstrap starting.
I20260929 16:41:19.530961 1842 tablet_bootstrap.cc:492] T a8102f42e7874aec9b3c55678a691cc2 P 969389c1869148b7ba94d01f14de514d: No bootstrap required, opened a new log
I20260929 16:41:19.531316 1842 ts_tablet_manager.cc:1397] T a8102f42e7874aec9b3c55678a691cc2 P 969389c1869148b7ba94d01f14de514d: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.003s
I20260929 16:41:19.534494 1845 tablet_bootstrap.cc:654] T a8102f42e7874aec9b3c55678a691cc2 P 659d0a10b5354643b948f00aa0297daf: Neither blocks nor log segments found. Creating new log.
I20260929 16:41:19.536405 1845 log.cc:824] T a8102f42e7874aec9b3c55678a691cc2 P 659d0a10b5354643b948f00aa0297daf: Log is configured to *not* fsync() on all Append() calls
I20260929 16:41:19.538829 1841 raft_consensus.cc:348] T a8102f42e7874aec9b3c55678a691cc2 P 7d549fb650c845e39af8c1ccf453b3b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d549fb650c845e39af8c1ccf453b3b1" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 35149 } } peers { permanent_uuid: "969389c1869148b7ba94d01f14de514d" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44753 } } peers { permanent_uuid: "659d0a10b5354643b948f00aa0297daf" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 38981 } }
I20260929 16:41:19.539433 1841 raft_consensus.cc:374] T a8102f42e7874aec9b3c55678a691cc2 P 7d549fb650c845e39af8c1ccf453b3b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:41:19.539551 1841 raft_consensus.cc:729] T a8102f42e7874aec9b3c55678a691cc2 P 7d549fb650c845e39af8c1ccf453b3b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d549fb650c845e39af8c1ccf453b3b1, State: Initialized, Role: FOLLOWER
I20260929 16:41:19.540330 1845 tablet_bootstrap.cc:492] T a8102f42e7874aec9b3c55678a691cc2 P 659d0a10b5354643b948f00aa0297daf: No bootstrap required, opened a new log
I20260929 16:41:19.540303 1841 consensus_queue.cc:260] T a8102f42e7874aec9b3c55678a691cc2 P 7d549fb650c845e39af8c1ccf453b3b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d549fb650c845e39af8c1ccf453b3b1" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 35149 } } peers { permanent_uuid: "969389c1869148b7ba94d01f14de514d" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44753 } } peers { permanent_uuid: "659d0a10b5354643b948f00aa0297daf" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 38981 } }
I20260929 16:41:19.540731 1845 ts_tablet_manager.cc:1397] T a8102f42e7874aec9b3c55678a691cc2 P 659d0a10b5354643b948f00aa0297daf: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.007s
I20260929 16:41:19.545548 1842 raft_consensus.cc:348] T a8102f42e7874aec9b3c55678a691cc2 P 969389c1869148b7ba94d01f14de514d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d549fb650c845e39af8c1ccf453b3b1" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 35149 } } peers { permanent_uuid: "969389c1869148b7ba94d01f14de514d" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44753 } } peers { permanent_uuid: "659d0a10b5354643b948f00aa0297daf" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 38981 } }
I20260929 16:41:19.546149 1842 raft_consensus.cc:374] T a8102f42e7874aec9b3c55678a691cc2 P 969389c1869148b7ba94d01f14de514d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:41:19.546202 1822 heartbeater.cc:503] Master 127.0.197.126:44117 was elected leader, sending a full tablet report...
I20260929 16:41:19.546263 1842 raft_consensus.cc:729] T a8102f42e7874aec9b3c55678a691cc2 P 969389c1869148b7ba94d01f14de514d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 969389c1869148b7ba94d01f14de514d, State: Initialized, Role: FOLLOWER
I20260929 16:41:19.547224 1841 ts_tablet_manager.cc:1428] T a8102f42e7874aec9b3c55678a691cc2 P 7d549fb650c845e39af8c1ccf453b3b1: Time spent starting tablet: real 0.021s user 0.014s sys 0.007s
I20260929 16:41:19.547197 1842 consensus_queue.cc:260] T a8102f42e7874aec9b3c55678a691cc2 P 969389c1869148b7ba94d01f14de514d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d549fb650c845e39af8c1ccf453b3b1" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 35149 } } peers { permanent_uuid: "969389c1869148b7ba94d01f14de514d" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44753 } } peers { permanent_uuid: "659d0a10b5354643b948f00aa0297daf" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 38981 } }
I20260929 16:41:19.549613 1842 ts_tablet_manager.cc:1428] T a8102f42e7874aec9b3c55678a691cc2 P 969389c1869148b7ba94d01f14de514d: Time spent starting tablet: real 0.018s user 0.017s sys 0.000s
I20260929 16:41:19.552300 1845 raft_consensus.cc:348] T a8102f42e7874aec9b3c55678a691cc2 P 659d0a10b5354643b948f00aa0297daf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d549fb650c845e39af8c1ccf453b3b1" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 35149 } } peers { permanent_uuid: "969389c1869148b7ba94d01f14de514d" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44753 } } peers { permanent_uuid: "659d0a10b5354643b948f00aa0297daf" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 38981 } }
I20260929 16:41:19.552912 1845 raft_consensus.cc:374] T a8102f42e7874aec9b3c55678a691cc2 P 659d0a10b5354643b948f00aa0297daf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:41:19.553040 1845 raft_consensus.cc:729] T a8102f42e7874aec9b3c55678a691cc2 P 659d0a10b5354643b948f00aa0297daf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 659d0a10b5354643b948f00aa0297daf, State: Initialized, Role: FOLLOWER
I20260929 16:41:19.553845 1845 consensus_queue.cc:260] T a8102f42e7874aec9b3c55678a691cc2 P 659d0a10b5354643b948f00aa0297daf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d549fb650c845e39af8c1ccf453b3b1" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 35149 } } peers { permanent_uuid: "969389c1869148b7ba94d01f14de514d" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44753 } } peers { permanent_uuid: "659d0a10b5354643b948f00aa0297daf" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 38981 } }
I20260929 16:41:19.555328 1688 heartbeater.cc:503] Master 127.0.197.126:44117 was elected leader, sending a full tablet report...
I20260929 16:41:19.555989 1845 ts_tablet_manager.cc:1428] T a8102f42e7874aec9b3c55678a691cc2 P 659d0a10b5354643b948f00aa0297daf: Time spent starting tablet: real 0.015s user 0.015s sys 0.001s
W20260929 16:41:19.593250 1689 tablet.cc:2369] T a8102f42e7874aec9b3c55678a691cc2 P 659d0a10b5354643b948f00aa0297daf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:41:19.636444 1823 tablet.cc:2369] T a8102f42e7874aec9b3c55678a691cc2 P 7d549fb650c845e39af8c1ccf453b3b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:41:19.761646 1849 raft_consensus.cc:482] T a8102f42e7874aec9b3c55678a691cc2 P 659d0a10b5354643b948f00aa0297daf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:41:19.762018 1849 raft_consensus.cc:504] T a8102f42e7874aec9b3c55678a691cc2 P 659d0a10b5354643b948f00aa0297daf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d549fb650c845e39af8c1ccf453b3b1" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 35149 } } peers { permanent_uuid: "969389c1869148b7ba94d01f14de514d" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44753 } } peers { permanent_uuid: "659d0a10b5354643b948f00aa0297daf" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 38981 } }
I20260929 16:41:19.764849 1849 leader_election.cc:288] T a8102f42e7874aec9b3c55678a691cc2 P 659d0a10b5354643b948f00aa0297daf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d549fb650c845e39af8c1ccf453b3b1 (127.0.197.67:35149), 969389c1869148b7ba94d01f14de514d (127.0.197.65:44753)
I20260929 16:41:19.773927 1836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:19.764516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:44311 (local address 127.0.197.67:35149)
0929 16:41:19.764763 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:19.764772 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:19.765721 (+ 949us) server_negotiation.cc:407] Connection header received
0929 16:41:19.765831 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:19.765835 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:19.765904 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:19.765993 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:41:19.766804 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:19.768003 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:19.771480 (+ 3477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:19.772788 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:19.772982 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:19.773064 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:19.773496 (+ 432us) server_negotiation.cc:299] Negotiation successful
0929 16:41:19.773660 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 16:41:19.774058 1838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:19.764628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:33137 (local address 127.0.197.65:44753)
0929 16:41:19.764819 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:19.764825 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:19.765554 (+ 729us) server_negotiation.cc:407] Connection header received
0929 16:41:19.765790 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:19.765797 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:19.765878 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:19.765993 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:41:19.766804 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:19.768003 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:19.771436 (+ 3433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:19.772788 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:19.772982 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:19.773070 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:19.773664 (+ 594us) server_negotiation.cc:299] Negotiation successful
0929 16:41:19.773818 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 16:41:19.774161 1850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:19.764265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:44753 (local address 127.0.197.66:33137)
0929 16:41:19.765404 (+ 1139us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:19.765434 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:19.765618 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:19.766031 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:19.766036 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:19.766068 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:41:19.766611 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:19.766630 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:19.768189 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:19.768195 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:19.771078 (+ 2883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:19.771100 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:19.773235 (+ 2135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:19.773240 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:19.773386 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:19.773454 (+ 68us) client_negotiation.cc:769] Sending connection context
0929 16:41:19.773614 (+ 160us) client_negotiation.cc:240] Negotiation successful
0929 16:41:19.773749 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":970,"mutex_wait_us":187,"thread_start_us":764,"threads_started":1}
I20260929 16:41:19.774533 1851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:19.764818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.67:35149 (local address 127.0.197.66:44311)
0929 16:41:19.765598 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:19.765613 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:19.765722 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:19.766031 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:19.766036 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:19.766092 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:41:19.766638 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:19.766645 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:19.768189 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:19.768195 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:19.771078 (+ 2883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:19.771100 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:19.773048 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:19.773054 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:19.773306 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:19.773338 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 16:41:19.773425 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 16:41:19.773566 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"mutex_wait_us":202,"spinlock_wait_cycles":36224,"thread_start_us":443,"threads_started":1}
I20260929 16:41:19.775308 1509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8102f42e7874aec9b3c55678a691cc2" candidate_uuid: "659d0a10b5354643b948f00aa0297daf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "969389c1869148b7ba94d01f14de514d" is_pre_election: true
I20260929 16:41:19.775729 1777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8102f42e7874aec9b3c55678a691cc2" candidate_uuid: "659d0a10b5354643b948f00aa0297daf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d549fb650c845e39af8c1ccf453b3b1" is_pre_election: true
I20260929 16:41:19.775954 1509 raft_consensus.cc:2454] T a8102f42e7874aec9b3c55678a691cc2 P 969389c1869148b7ba94d01f14de514d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 659d0a10b5354643b948f00aa0297daf in term 0.
I20260929 16:41:19.776248 1777 raft_consensus.cc:2454] T a8102f42e7874aec9b3c55678a691cc2 P 7d549fb650c845e39af8c1ccf453b3b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 659d0a10b5354643b948f00aa0297daf in term 0.
I20260929 16:41:19.776767 1577 leader_election.cc:302] T a8102f42e7874aec9b3c55678a691cc2 P 659d0a10b5354643b948f00aa0297daf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 659d0a10b5354643b948f00aa0297daf, 969389c1869148b7ba94d01f14de514d; no voters:
I20260929 16:41:19.777257 1849 raft_consensus.cc:2781] T a8102f42e7874aec9b3c55678a691cc2 P 659d0a10b5354643b948f00aa0297daf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:41:19.777851 1849 raft_consensus.cc:482] T a8102f42e7874aec9b3c55678a691cc2 P 659d0a10b5354643b948f00aa0297daf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:41:19.777963 1849 raft_consensus.cc:3037] T a8102f42e7874aec9b3c55678a691cc2 P 659d0a10b5354643b948f00aa0297daf [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:41:19.780501 1849 raft_consensus.cc:504] T a8102f42e7874aec9b3c55678a691cc2 P 659d0a10b5354643b948f00aa0297daf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d549fb650c845e39af8c1ccf453b3b1" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 35149 } } peers { permanent_uuid: "969389c1869148b7ba94d01f14de514d" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44753 } } peers { permanent_uuid: "659d0a10b5354643b948f00aa0297daf" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 38981 } }
I20260929 16:41:19.781199 1849 leader_election.cc:288] T a8102f42e7874aec9b3c55678a691cc2 P 659d0a10b5354643b948f00aa0297daf [CANDIDATE]: Term 1 election: Requested vote from peers 7d549fb650c845e39af8c1ccf453b3b1 (127.0.197.67:35149), 969389c1869148b7ba94d01f14de514d (127.0.197.65:44753)
I20260929 16:41:19.781788 1777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8102f42e7874aec9b3c55678a691cc2" candidate_uuid: "659d0a10b5354643b948f00aa0297daf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d549fb650c845e39af8c1ccf453b3b1"
I20260929 16:41:19.781790 1509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8102f42e7874aec9b3c55678a691cc2" candidate_uuid: "659d0a10b5354643b948f00aa0297daf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "969389c1869148b7ba94d01f14de514d"
I20260929 16:41:19.782020 1777 raft_consensus.cc:3037] T a8102f42e7874aec9b3c55678a691cc2 P 7d549fb650c845e39af8c1ccf453b3b1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:41:19.782034 1509 raft_consensus.cc:3037] T a8102f42e7874aec9b3c55678a691cc2 P 969389c1869148b7ba94d01f14de514d [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:41:19.784476 1777 raft_consensus.cc:2454] T a8102f42e7874aec9b3c55678a691cc2 P 7d549fb650c845e39af8c1ccf453b3b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 659d0a10b5354643b948f00aa0297daf in term 1.
I20260929 16:41:19.784571 1509 raft_consensus.cc:2454] T a8102f42e7874aec9b3c55678a691cc2 P 969389c1869148b7ba94d01f14de514d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 659d0a10b5354643b948f00aa0297daf in term 1.
I20260929 16:41:19.785128 1576 leader_election.cc:302] T a8102f42e7874aec9b3c55678a691cc2 P 659d0a10b5354643b948f00aa0297daf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 659d0a10b5354643b948f00aa0297daf, 7d549fb650c845e39af8c1ccf453b3b1, 969389c1869148b7ba94d01f14de514d; no voters:
I20260929 16:41:19.785562 1849 raft_consensus.cc:2781] T a8102f42e7874aec9b3c55678a691cc2 P 659d0a10b5354643b948f00aa0297daf [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:41:19.786046 1849 raft_consensus.cc:686] T a8102f42e7874aec9b3c55678a691cc2 P 659d0a10b5354643b948f00aa0297daf [term 1 LEADER]: Becoming Leader. State: Replica: 659d0a10b5354643b948f00aa0297daf, State: Running, Role: LEADER
I20260929 16:41:19.786814 1849 consensus_queue.cc:237] T a8102f42e7874aec9b3c55678a691cc2 P 659d0a10b5354643b948f00aa0297daf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d549fb650c845e39af8c1ccf453b3b1" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 35149 } } peers { permanent_uuid: "969389c1869148b7ba94d01f14de514d" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44753 } } peers { permanent_uuid: "659d0a10b5354643b948f00aa0297daf" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 38981 } }
I20260929 16:41:19.792111 1365 catalog_manager.cc:5495] T a8102f42e7874aec9b3c55678a691cc2 P 659d0a10b5354643b948f00aa0297daf reported cstate change: term changed from 0 to 1, leader changed from <none> to 659d0a10b5354643b948f00aa0297daf (127.0.197.66). New cstate: current_term: 1 leader_uuid: "659d0a10b5354643b948f00aa0297daf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d549fb650c845e39af8c1ccf453b3b1" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 35149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "969389c1869148b7ba94d01f14de514d" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "659d0a10b5354643b948f00aa0297daf" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 38981 } health_report { overall_health: HEALTHY } } }
W20260929 16:41:19.796413 1555 tablet.cc:2369] T a8102f42e7874aec9b3c55678a691cc2 P 969389c1869148b7ba94d01f14de514d: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 16:41:19 dist-test-slave-l31f krb5kdc[1320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.65@KRBTEST.COM
I20260929 16:41:19.867223 1838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:19.850199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59966 (local address 127.0.197.65:44753)
0929 16:41:19.850466 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:19.850476 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:19.850888 (+ 412us) server_negotiation.cc:407] Connection header received
0929 16:41:19.851187 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:19.851198 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:19.851269 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:19.851390 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:19.853497 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:19.854303 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:19.855806 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:19.856288 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:19.856455 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:19.860984 (+ 4529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:19.861002 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:19.861019 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:19.861068 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:19.864182 (+ 3114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:19.865070 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:19.865080 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:19.865082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:19.865246 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:19.865693 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:19.865696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:19.865699 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:19.866087 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:19.866340 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:19.866616 (+ 276us) server_negotiation.cc:299] Negotiation successful
0929 16:41:19.866807 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
Sep 29 16:41:19 dist-test-slave-l31f krb5kdc[1320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700079, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 16:41:19 dist-test-slave-l31f krb5kdc[1320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700079, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.197.65@KRBTEST.COM
I20260929 16:41:19.919189 1838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:19.903579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59974 (local address 127.0.197.65:44753)
0929 16:41:19.903822 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:19.903833 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:19.904845 (+ 1012us) server_negotiation.cc:407] Connection header received
0929 16:41:19.905087 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:19.905092 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:19.905164 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:19.905278 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:19.906665 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:19.907553 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:19.908974 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:19.909444 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:19.909564 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:19.913901 (+ 4337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:19.913907 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:19.913911 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:19.913951 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:19.916721 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:19.917404 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:19.917408 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:19.917410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:19.917533 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:19.917881 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:19.917884 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:19.917886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:19.918156 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:19.918361 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:19.918591 (+ 230us) server_negotiation.cc:299] Negotiation successful
0929 16:41:19.918760 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
Sep 29 16:41:19 dist-test-slave-l31f krb5kdc[1320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 16:41:19 dist-test-slave-l31f krb5kdc[1320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.65@KRBTEST.COM
I20260929 16:41:19.968820 1838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:19.952982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59984 (local address 127.0.197.65:44753)
0929 16:41:19.953186 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:19.953193 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:19.953593 (+ 400us) server_negotiation.cc:407] Connection header received
0929 16:41:19.953819 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:19.953824 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:19.953889 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:19.954001 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:19.955465 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:19.956188 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:19.957476 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:19.957920 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:19.958070 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:19.962750 (+ 4680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:19.962756 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:19.962760 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:19.962811 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:19.965545 (+ 2734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:19.966495 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:19.966500 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:19.966502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:19.966659 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:19.967115 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:19.967120 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:19.967122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:19.967465 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:19.967676 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:19.968197 (+ 521us) server_negotiation.cc:299] Negotiation successful
0929 16:41:19.968545 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260929 16:41:19.970839 1469 tablet_service.cc:2908] Rejecting scan request for tablet a8102f42e7874aec9b3c55678a691cc2: Uninitialized: clean time has not yet been initialized
I20260929 16:41:19.974175 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 1426
I20260929 16:41:19.987941 1838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:19.975569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59990 (local address 127.0.197.65:44753)
0929 16:41:19.975734 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:19.975739 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:19.976217 (+ 478us) server_negotiation.cc:407] Connection header received
0929 16:41:19.976339 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:19.976343 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:19.976426 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:19.976513 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:19.977674 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:19.978370 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:19.979739 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:19.980168 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:19.980289 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:19.982544 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:19.982550 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:19.982553 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:19.982589 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:19.985311 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:19.986084 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:19.986088 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:19.986090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:19.986221 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:19.986564 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:19.986567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:19.986569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:19.986861 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:19.987206 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:19.987518 (+ 312us) server_negotiation.cc:299] Negotiation successful
0929 16:41:19.987701 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 16:41:19.988754 1549 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:41:20.082777 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 1426
I20260929 16:41:20.101011 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 1560
Sep 29 16:41:20 dist-test-slave-l31f krb5kdc[1320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.66@KRBTEST.COM
I20260929 16:41:20.117777 1866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:20.101822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50266 (local address 127.0.197.66:38981)
0929 16:41:20.102355 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:20.102360 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:20.102378 (+ 18us) server_negotiation.cc:407] Connection header received
0929 16:41:20.102433 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:20.102436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:20.102505 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:20.102601 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:20.103953 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:20.104662 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:20.106094 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:20.106516 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:20.106623 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:20.111178 (+ 4555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:20.111196 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:20.111213 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:20.111257 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:20.114320 (+ 3063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:20.115252 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:20.115261 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:20.115264 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:20.115414 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:20.115890 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:20.115894 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:20.115896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:20.116309 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:20.116573 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:20.117202 (+ 629us) server_negotiation.cc:299] Negotiation successful
0929 16:41:20.117401 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":301,"threads_started":1}
I20260929 16:41:20.118465 1683 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:41:20.211104 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 1560
I20260929 16:41:20.228778 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 1694
Sep 29 16:41:20 dist-test-slave-l31f krb5kdc[1320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.67@KRBTEST.COM
I20260929 16:41:20.249384 1836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:20.229681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55792 (local address 127.0.197.67:35149)
0929 16:41:20.229873 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:20.229879 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:20.229900 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:41:20.230083 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:20.230089 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:20.230162 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:20.230286 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:20.231598 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:20.232324 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:20.233657 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:20.234067 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:20.234211 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:20.242876 (+ 8665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:20.242895 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:20.242913 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:20.242958 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:20.245997 (+ 3039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:20.246814 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:20.246821 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:20.246824 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:20.246993 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:20.247440 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:20.247444 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:20.247446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:20.247807 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:20.248034 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:20.248581 (+ 547us) server_negotiation.cc:299] Negotiation successful
0929 16:41:20.249086 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 16:41:20.250170 1817 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:41:20.341259 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 1694
I20260929 16:41:20.359625 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 1333
I20260929 16:41:20.361227 1395 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:41:20.469810 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 1333
2026-09-29T16:41:20Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (7917 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:41:20 dist-test-slave-l31f krb5kdc[1872](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:41:20 dist-test-slave-l31f krb5kdc[1872](info): set up 2 sockets
Sep 29 16:41:20 dist-test-slave-l31f krb5kdc[1872](info): commencing operation
krb5kdc: starting...
W20260929 16:41:21.587141 789 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/krb5kdc/test-user.keytab.
Sep 29 16:41:21 dist-test-slave-l31f krb5kdc[1872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:41:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:41:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.197.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
I20260929 16:41:21.795349 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:37717
--webserver_interface=127.0.197.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:38813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.126:37717
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:41:22.147341 1885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:41:22.147734 1885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:41:22.147912 1885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:41:22.155712 1885 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:41:22.155897 1885 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 16:41:22.156001 1885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:41:22.156126 1885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:41:22.156214 1885 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:41:22.156304 1885 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:41:22.167948 1885 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:38813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.197.126:37717
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:37717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.197.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:41:22.169484 1885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:41:22.170975 1885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:41:22.183902 1890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:22.183975 1891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:22.184122 1893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:41:22.185520 1885 server_base.cc:1034] running on GCE node
I20260929 16:41:22.186514 1885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:41:22.188226 1885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:41:22.189520 1885 hybrid_clock.cc:648] HybridClock initialized: now 1790700082189438 us; error 61 us; skew 500 ppm
I20260929 16:41:22.189901 1885 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:41:22 dist-test-slave-l31f krb5kdc[1872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:41:22.193279 1885 init.cc:377] Logged in from keytab as kudu/127.0.197.126@KRBTEST.COM (short username kudu)
I20260929 16:41:22.195161 1885 webserver.cc:460] Webserver started at http://127.0.197.126:43291/ using document root <none> and password file <none>
I20260929 16:41:22.196118 1885 fs_manager.cc:359] Metadata directory not provided
I20260929 16:41:22.196259 1885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:41:22.196621 1885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:41:22.199945 1885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/master-0/data/instance:
uuid: "b1e9d1fcf7c2470883bd4ca247fff193"
format_stamp: "Formatted at 2026-09-29 16:41:22 on dist-test-slave-l31f"
server_key: "96e4ba103a0948078ac093a394abb319"
server_key_iv: "8891bed794598797d5978a36f1fb089b"
server_key_version: "encryptionkey@0"
I20260929 16:41:22.200867 1885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/master-0/wal/instance:
uuid: "b1e9d1fcf7c2470883bd4ca247fff193"
format_stamp: "Formatted at 2026-09-29 16:41:22 on dist-test-slave-l31f"
server_key: "96e4ba103a0948078ac093a394abb319"
server_key_iv: "8891bed794598797d5978a36f1fb089b"
server_key_version: "encryptionkey@0"
I20260929 16:41:22.206871 1885 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 16:41:22.211200 1900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:22.212829 1885 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260929 16:41:22.213107 1885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "b1e9d1fcf7c2470883bd4ca247fff193"
format_stamp: "Formatted at 2026-09-29 16:41:22 on dist-test-slave-l31f"
server_key: "96e4ba103a0948078ac093a394abb319"
server_key_iv: "8891bed794598797d5978a36f1fb089b"
server_key_version: "encryptionkey@0"
I20260929 16:41:22.213361 1885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:41:22.273749 1885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:41:22.278481 1885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:41:22.278898 1885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:41:22.301723 1885 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.126:37717
I20260929 16:41:22.301767 1961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.126:37717 every 8 connection(s)
I20260929 16:41:22.303275 1885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:41:22.306915 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 1885
I20260929 16:41:22.307257 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/master-0/wal/instance
I20260929 16:41:22.307308 1962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:41:22.307893 789 external_mini_cluster.cc:1428] Setting key bcce903a1023622da0eab989be819933
I20260929 16:41:22.323233 1962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1e9d1fcf7c2470883bd4ca247fff193: Bootstrap starting.
Sep 29 16:41:22 dist-test-slave-l31f krb5kdc[1872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:41:22.327517 1962 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1e9d1fcf7c2470883bd4ca247fff193: Neither blocks nor log segments found. Creating new log.
I20260929 16:41:22.328995 1962 log.cc:824] T 00000000000000000000000000000000 P b1e9d1fcf7c2470883bd4ca247fff193: Log is configured to *not* fsync() on all Append() calls
I20260929 16:41:22.333133 1962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1e9d1fcf7c2470883bd4ca247fff193: No bootstrap required, opened a new log
I20260929 16:41:22.334693 1965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:22.309921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44460 (local address 127.0.197.126:37717)
0929 16:41:22.310858 (+ 937us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:22.310870 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:22.310918 (+ 48us) server_negotiation.cc:407] Connection header received
0929 16:41:22.312690 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:22.312715 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:22.313175 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:22.313558 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:22.315594 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:22.317073 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:22.319202 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:22.320035 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:22.320210 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:22.325199 (+ 4989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:22.325215 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:22.325239 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:22.325297 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:22.329399 (+ 4102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:22.330521 (+ 1122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:22.330528 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:22.330535 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:22.330802 (+ 267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:22.331306 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:22.331312 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:22.331315 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:22.332062 (+ 747us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:22.332391 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:22.332800 (+ 409us) server_negotiation.cc:299] Negotiation successful
0929 16:41:22.333299 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":715,"thread_start_us":487,"threads_started":1}
I20260929 16:41:22.343758 1962 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1e9d1fcf7c2470883bd4ca247fff193 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1e9d1fcf7c2470883bd4ca247fff193" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 37717 } }
I20260929 16:41:22.344189 1962 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1e9d1fcf7c2470883bd4ca247fff193 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:41:22.344266 1962 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1e9d1fcf7c2470883bd4ca247fff193 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1e9d1fcf7c2470883bd4ca247fff193, State: Initialized, Role: FOLLOWER
I20260929 16:41:22.344805 1962 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1e9d1fcf7c2470883bd4ca247fff193 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1e9d1fcf7c2470883bd4ca247fff193" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 37717 } }
I20260929 16:41:22.345022 1962 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1e9d1fcf7c2470883bd4ca247fff193 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:41:22.345122 1962 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1e9d1fcf7c2470883bd4ca247fff193 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:41:22.345253 1962 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1e9d1fcf7c2470883bd4ca247fff193 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:41:22.347848 1962 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1e9d1fcf7c2470883bd4ca247fff193 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1e9d1fcf7c2470883bd4ca247fff193" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 37717 } }
I20260929 16:41:22.348357 1962 leader_election.cc:302] T 00000000000000000000000000000000 P b1e9d1fcf7c2470883bd4ca247fff193 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1e9d1fcf7c2470883bd4ca247fff193; no voters:
I20260929 16:41:22.348903 1962 leader_election.cc:288] T 00000000000000000000000000000000 P b1e9d1fcf7c2470883bd4ca247fff193 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:41:22.349107 1967 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1e9d1fcf7c2470883bd4ca247fff193 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:41:22.349864 1967 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1e9d1fcf7c2470883bd4ca247fff193 [term 1 LEADER]: Becoming Leader. State: Replica: b1e9d1fcf7c2470883bd4ca247fff193, State: Running, Role: LEADER
I20260929 16:41:22.350427 1967 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1e9d1fcf7c2470883bd4ca247fff193 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1e9d1fcf7c2470883bd4ca247fff193" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 37717 } }
I20260929 16:41:22.351212 1962 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1e9d1fcf7c2470883bd4ca247fff193 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:41:22.355146 1969 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1e9d1fcf7c2470883bd4ca247fff193 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1e9d1fcf7c2470883bd4ca247fff193. Latest consensus state: current_term: 1 leader_uuid: "b1e9d1fcf7c2470883bd4ca247fff193" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1e9d1fcf7c2470883bd4ca247fff193" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 37717 } } }
I20260929 16:41:22.355472 1969 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1e9d1fcf7c2470883bd4ca247fff193 [sys.catalog]: This master's current role is: LEADER
I20260929 16:41:22.353237 1968 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1e9d1fcf7c2470883bd4ca247fff193 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1e9d1fcf7c2470883bd4ca247fff193" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1e9d1fcf7c2470883bd4ca247fff193" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 37717 } } }
I20260929 16:41:22.357007 1968 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1e9d1fcf7c2470883bd4ca247fff193 [sys.catalog]: This master's current role is: LEADER
I20260929 16:41:22.358588 1973 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:41:22.363657 1973 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:41:22.372733 1973 catalog_manager.cc:1347] Generated new cluster ID: 3278e71069f54a7c922220ecb649ed68
I20260929 16:41:22.372874 1973 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:41:22.381532 1973 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:41:22.383302 1973 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:41:22.390792 1973 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1e9d1fcf7c2470883bd4ca247fff193: Generated new TSK 0
I20260929 16:41:22.392067 1973 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.197.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab.
Entry for principal kudu/127.0.197.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab.
I20260929 16:41:22.448525 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.65:0
--local_ip_for_outbound_sockets=127.0.197.65
--webserver_interface=127.0.197.65
--webserver_port=0
--tserver_master_addrs=127.0.197.126:37717
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab
--principal=kudu/127.0.197.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:38813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:41:22.813169 1988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:41:22.813483 1988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:41:22.813657 1988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:41:22.821671 1988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:41:22.821815 1988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:41:22.821956 1988 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 16:41:22.822099 1988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.65
I20260929 16:41:22.833191 1988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:38813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab
--principal=kudu/127.0.197.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.197.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.197.126:37717
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.65
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:41:22.834955 1988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:41:22.836645 1988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:41:22.851670 1994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:22.851599 1996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:22.851807 1993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:41:22.852641 1988 server_base.cc:1034] running on GCE node
I20260929 16:41:22.853343 1988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:41:22.854430 1988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:41:22.855806 1988 hybrid_clock.cc:648] HybridClock initialized: now 1790700082855731 us; error 167 us; skew 500 ppm
I20260929 16:41:22.856240 1988 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:41:22 dist-test-slave-l31f krb5kdc[1872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:41:22.860430 1988 init.cc:377] Logged in from keytab as kudu/127.0.197.65@KRBTEST.COM (short username kudu)
I20260929 16:41:22.862452 1988 webserver.cc:460] Webserver started at http://127.0.197.65:43517/ using document root <none> and password file <none>
I20260929 16:41:22.863453 1988 fs_manager.cc:359] Metadata directory not provided
I20260929 16:41:22.863572 1988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:41:22.864030 1988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:41:22.867632 1988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/ts-0/data/instance:
uuid: "f0a80dcdaf6d4a299f3aefbd155e3783"
format_stamp: "Formatted at 2026-09-29 16:41:22 on dist-test-slave-l31f"
server_key: "dadd65f90e326d044368247af162e1e6"
server_key_iv: "f48ed59820d85616c4d9eda5d32c61a2"
server_key_version: "encryptionkey@0"
I20260929 16:41:22.868687 1988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/ts-0/wal/instance:
uuid: "f0a80dcdaf6d4a299f3aefbd155e3783"
format_stamp: "Formatted at 2026-09-29 16:41:22 on dist-test-slave-l31f"
server_key: "dadd65f90e326d044368247af162e1e6"
server_key_iv: "f48ed59820d85616c4d9eda5d32c61a2"
server_key_version: "encryptionkey@0"
I20260929 16:41:22.874958 1988 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 16:41:22.879568 2003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:22.881820 1988 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:41:22.882277 1988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/ts-0/wal
uuid: "f0a80dcdaf6d4a299f3aefbd155e3783"
format_stamp: "Formatted at 2026-09-29 16:41:22 on dist-test-slave-l31f"
server_key: "dadd65f90e326d044368247af162e1e6"
server_key_iv: "f48ed59820d85616c4d9eda5d32c61a2"
server_key_version: "encryptionkey@0"
I20260929 16:41:22.882733 1988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:41:22.905361 1988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:41:22.910588 1988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:41:22.911242 1988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:41:22.914381 1988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:41:22.914521 1988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:22.914703 1988 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:41:22.914783 1988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:41:22 dist-test-slave-l31f krb5kdc[1872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.65@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:41:22.941638 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:22.915296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:55975 (local address 127.0.197.126:37717)
0929 16:41:22.917994 (+ 2698us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:22.918001 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:22.918029 (+ 28us) server_negotiation.cc:407] Connection header received
0929 16:41:22.919797 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:22.919803 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:22.919903 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:22.920057 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:22.922582 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:22.923597 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:22.926406 (+ 2809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:22.926894 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:22.927105 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:22.931654 (+ 4549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:22.931664 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:22.931671 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:22.931745 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:22.936031 (+ 4286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:22.937828 (+ 1797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:22.937840 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:22.937843 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:22.938121 (+ 278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:22.939114 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:22.939121 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:22.939130 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:22.939466 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:22.939668 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:22.941111 (+ 1443us) server_negotiation.cc:299] Negotiation successful
0929 16:41:22.941341 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2559,"thread_start_us":2458,"threads_started":1}
I20260929 16:41:22.942878 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:22.915997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37717 (local address 127.0.197.65:55975)
0929 16:41:22.916976 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:22.917030 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:22.919375 (+ 2345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:22.920352 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:22.920362 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:22.921248 (+ 886us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:41:22.922334 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:22.922361 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:22.923773 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:22.923777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:22.926210 (+ 2433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:22.926232 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:22.927220 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:22.927228 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:22.927551 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:22.928606 (+ 1055us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:41:22.928631 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:41:22.931301 (+ 2670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:41:22.936331 (+ 5030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:22.936343 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:22.936371 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:22.937605 (+ 1234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:22.938351 (+ 746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:22.938371 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:22.938378 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:22.938904 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:22.939660 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:41:22.939670 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:41:22.940443 (+ 773us) client_negotiation.cc:769] Sending connection context
0929 16:41:22.940930 (+ 487us) client_negotiation.cc:240] Negotiation successful
0929 16:41:22.941494 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":429,"threads_started":1}
I20260929 16:41:22.972242 1988 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.65:38669
I20260929 16:41:22.972199 2123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.65:38669 every 8 connection(s)
I20260929 16:41:22.973914 1988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
I20260929 16:41:22.975370 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:22.952660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:40269 (local address 127.0.197.126:37717)
0929 16:41:22.952938 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:22.952948 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:22.955975 (+ 3027us) server_negotiation.cc:407] Connection header received
0929 16:41:22.956431 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:22.956438 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:22.956556 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:22.956844 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:22.958747 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:22.960082 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:22.962852 (+ 2770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:22.963891 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:22.964072 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:22.967643 (+ 3571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:22.967652 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:22.967660 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:22.967719 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:22.971733 (+ 4014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:22.972873 (+ 1140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:22.972906 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:22.972909 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:22.973059 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:22.973515 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:22.973520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:22.973522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:22.973839 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:22.974021 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:22.974596 (+ 575us) server_negotiation.cc:299] Negotiation successful
0929 16:41:22.974919 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260929 16:41:22.975682 2086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:22.952627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37717 (local address 127.0.197.65:40269)
0929 16:41:22.955757 (+ 3130us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:22.955790 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:22.956053 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:22.956935 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:22.956941 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:22.957826 (+ 885us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:41:22.958462 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:22.958486 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:22.960383 (+ 1897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:22.960392 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:22.962595 (+ 2203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:22.962616 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:22.964202 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:22.964214 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:22.964483 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:22.965463 (+ 980us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:41:22.965502 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:41:22.967299 (+ 1797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:41:22.971992 (+ 4693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:22.971999 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:22.972003 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:22.972668 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:22.973206 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:22.973211 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:22.973214 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:22.973372 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:22.973989 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:41:22.973995 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:41:22.974217 (+ 222us) client_negotiation.cc:769] Sending connection context
0929 16:41:22.974403 (+ 186us) client_negotiation.cc:240] Negotiation successful
0929 16:41:22.974935 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2891,"thread_start_us":2655,"threads_started":1}
I20260929 16:41:22.977109 2124 heartbeater.cc:347] Connected to a master server at 127.0.197.126:37717
I20260929 16:41:22.977435 2124 heartbeater.cc:464] Registering TS with master...
I20260929 16:41:22.978359 2124 heartbeater.cc:511] Master 127.0.197.126:37717 requested a full tablet report, sending...
I20260929 16:41:22.981093 1917 ts_manager.cc:194] Registered new tserver with Master: f0a80dcdaf6d4a299f3aefbd155e3783 (127.0.197.65:38669)
I20260929 16:41:22.982121 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 1988
I20260929 16:41:22.982340 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/ts-0/wal/instance
I20260929 16:41:22.982908 1917 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.65@KRBTEST.COM'} at 127.0.197.65:55975
I20260929 16:41:22.982981 789 external_mini_cluster.cc:1428] Setting key f0f74fd32418472e69420e50db48cbcc
WARNING: no policy specified for kudu/127.0.197.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab.
Entry for principal kudu/127.0.197.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab.
I20260929 16:41:23.032586 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.66:0
--local_ip_for_outbound_sockets=127.0.197.66
--webserver_interface=127.0.197.66
--webserver_port=0
--tserver_master_addrs=127.0.197.126:37717
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab
--principal=kudu/127.0.197.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:38813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:41:23.386281 2128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:41:23.386605 2128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:41:23.386775 2128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:41:23.394677 2128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:41:23.394824 2128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:41:23.394980 2128 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 16:41:23.395151 2128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.66
I20260929 16:41:23.405620 2128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:38813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab
--principal=kudu/127.0.197.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.197.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.197.126:37717
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.66
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:41:23.407117 2128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:41:23.408548 2128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:41:23.426482 2134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:23.426441 2133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:23.426525 2136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:41:23.428544 2128 server_base.cc:1034] running on GCE node
I20260929 16:41:23.429246 2128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:41:23.430235 2128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:41:23.431504 2128 hybrid_clock.cc:648] HybridClock initialized: now 1790700083431416 us; error 40 us; skew 500 ppm
I20260929 16:41:23.431876 2128 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:41:23 dist-test-slave-l31f krb5kdc[1872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:41:23.435472 2128 init.cc:377] Logged in from keytab as kudu/127.0.197.66@KRBTEST.COM (short username kudu)
I20260929 16:41:23.437325 2128 webserver.cc:460] Webserver started at http://127.0.197.66:38757/ using document root <none> and password file <none>
I20260929 16:41:23.438123 2128 fs_manager.cc:359] Metadata directory not provided
I20260929 16:41:23.438273 2128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:41:23.438634 2128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:41:23.441838 2128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/ts-1/data/instance:
uuid: "46bb3be715984c2aa30664a8fb35cf7b"
format_stamp: "Formatted at 2026-09-29 16:41:23 on dist-test-slave-l31f"
server_key: "055c37c433a90cf1c6a0a5d01ff39161"
server_key_iv: "cad129d8268533955960d1774d76b5a3"
server_key_version: "encryptionkey@0"
I20260929 16:41:23.442715 2128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/ts-1/wal/instance:
uuid: "46bb3be715984c2aa30664a8fb35cf7b"
format_stamp: "Formatted at 2026-09-29 16:41:23 on dist-test-slave-l31f"
server_key: "055c37c433a90cf1c6a0a5d01ff39161"
server_key_iv: "cad129d8268533955960d1774d76b5a3"
server_key_version: "encryptionkey@0"
I20260929 16:41:23.449317 2128 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260929 16:41:23.453768 2143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:23.455938 2128 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:41:23.456198 2128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/ts-1/wal
uuid: "46bb3be715984c2aa30664a8fb35cf7b"
format_stamp: "Formatted at 2026-09-29 16:41:23 on dist-test-slave-l31f"
server_key: "055c37c433a90cf1c6a0a5d01ff39161"
server_key_iv: "cad129d8268533955960d1774d76b5a3"
server_key_version: "encryptionkey@0"
I20260929 16:41:23.456468 2128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:41:23.491261 2128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:41:23.496107 2128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:41:23.496412 2128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:41:23.499317 2128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:41:23.499480 2128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:23.499667 2128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:41:23.499749 2128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:41:23 dist-test-slave-l31f krb5kdc[1872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.66@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:41:23.531116 2152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:23.500341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:35853 (local address 127.0.197.126:37717)
0929 16:41:23.500851 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:23.500860 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:23.502258 (+ 1398us) server_negotiation.cc:407] Connection header received
0929 16:41:23.505775 (+ 3517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:23.505784 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:23.505916 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:23.506025 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:23.509746 (+ 3721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:23.510837 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:23.514738 (+ 3901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:23.515499 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:23.515658 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:23.522167 (+ 6509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:23.522178 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:23.522186 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:23.522247 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:23.525598 (+ 3351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:23.527078 (+ 1480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:23.527086 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:23.527091 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:23.527287 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:23.528069 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:23.528075 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:23.528079 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:23.528539 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:23.528752 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:23.530179 (+ 1427us) server_negotiation.cc:299] Negotiation successful
0929 16:41:23.530530 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":239,"threads_started":1}
I20260929 16:41:23.533681 2153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:23.501090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37717 (local address 127.0.197.66:35853)
0929 16:41:23.502020 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:23.502085 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:23.505247 (+ 3162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:23.506753 (+ 1506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:23.506775 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:23.507979 (+ 1204us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:41:23.509361 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:23.509403 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:23.511345 (+ 1942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:23.511387 (+ 42us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:23.514480 (+ 3093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:23.514511 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:23.515863 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:23.515872 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:23.516306 (+ 434us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:23.517363 (+ 1057us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:41:23.517407 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:41:23.521738 (+ 4331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:41:23.525837 (+ 4099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:23.525854 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:23.525881 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:23.526831 (+ 950us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:23.527456 (+ 625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:23.527465 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:23.527474 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:23.527851 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:23.528773 (+ 922us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:41:23.528786 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:41:23.529539 (+ 753us) client_negotiation.cc:769] Sending connection context
0929 16:41:23.530033 (+ 494us) client_negotiation.cc:240] Negotiation successful
0929 16:41:23.530870 (+ 837us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":353,"threads_started":1}
I20260929 16:41:23.548477 2128 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.66:41639
I20260929 16:41:23.548519 2263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.66:41639 every 8 connection(s)
I20260929 16:41:23.549939 2128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
I20260929 16:41:23.552932 2264 heartbeater.cc:347] Connected to a master server at 127.0.197.126:37717
I20260929 16:41:23.553198 2264 heartbeater.cc:464] Registering TS with master...
I20260929 16:41:23.554004 2264 heartbeater.cc:511] Master 127.0.197.126:37717 requested a full tablet report, sending...
I20260929 16:41:23.554198 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 2128
I20260929 16:41:23.554407 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/ts-1/wal/instance
I20260929 16:41:23.554972 789 external_mini_cluster.cc:1428] Setting key 2f761dee198326dbec8a8ffa35d9bb4b
I20260929 16:41:23.555642 1917 ts_manager.cc:194] Registered new tserver with Master: 46bb3be715984c2aa30664a8fb35cf7b (127.0.197.66:41639)
I20260929 16:41:23.556979 1917 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.66@KRBTEST.COM'} at 127.0.197.66:35853
I20260929 16:41:23.559264 2254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:23.543244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37717 (local address 127.0.197.66:43255)
0929 16:41:23.544064 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:23.544089 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:23.544294 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:23.544889 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:23.544893 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:23.545369 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:41:23.545875 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:23.545893 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:23.547062 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:23.547070 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:23.548444 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:23.548462 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:23.549282 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:23.549289 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:23.549487 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:23.550195 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:41:23.550213 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:41:23.551463 (+ 1250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:41:23.555607 (+ 4144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:23.555613 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:23.555619 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:23.556287 (+ 668us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:23.556768 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:23.556772 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:23.556775 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:23.556945 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:23.557820 (+ 875us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:41:23.557826 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:41:23.558062 (+ 236us) client_negotiation.cc:769] Sending connection context
0929 16:41:23.558207 (+ 145us) client_negotiation.cc:240] Negotiation successful
0929 16:41:23.558616 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"thread_start_us":496,"threads_started":1}
I20260929 16:41:23.559755 2152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:23.543115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:43255 (local address 127.0.197.126:37717)
0929 16:41:23.543309 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:23.543314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:23.544200 (+ 886us) server_negotiation.cc:407] Connection header received
0929 16:41:23.544473 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:23.544481 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:23.544581 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:23.544693 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:23.546073 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:23.546874 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:23.548622 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:23.549078 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:23.549241 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:23.551707 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:23.551712 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:23.551716 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:23.551752 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:23.555413 (+ 3661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:23.556460 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:23.556468 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:23.556475 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:23.556650 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:23.557123 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:23.557131 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:23.557135 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:23.557549 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:23.557754 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:23.559222 (+ 1468us) server_negotiation.cc:299] Negotiation successful
0929 16:41:23.559468 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
WARNING: no policy specified for kudu/127.0.197.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab.
Entry for principal kudu/127.0.197.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab.
I20260929 16:41:23.603853 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.67:0
--local_ip_for_outbound_sockets=127.0.197.67
--webserver_interface=127.0.197.67
--webserver_port=0
--tserver_master_addrs=127.0.197.126:37717
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab
--principal=kudu/127.0.197.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:38813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:41:23.968918 2268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:41:23.969231 2268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:41:23.969419 2268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:41:23.977142 2268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:41:23.977272 2268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:41:23.977438 2268 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 16:41:23.977558 2268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.67
I20260929 16:41:23.986716 2124 heartbeater.cc:503] Master 127.0.197.126:37717 was elected leader, sending a full tablet report...
I20260929 16:41:23.988373 2268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:38813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab
--principal=kudu/127.0.197.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.197.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.197.126:37717
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.67
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:41:23.989886 2268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:41:23.991302 2268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:41:24.005317 2276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:24.005290 2273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:24.005257 2274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:41:24.006390 2268 server_base.cc:1034] running on GCE node
I20260929 16:41:24.007066 2268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:41:24.008136 2268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:41:24.009400 2268 hybrid_clock.cc:648] HybridClock initialized: now 1790700084009335 us; error 34 us; skew 500 ppm
I20260929 16:41:24.009719 2268 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:41:24 dist-test-slave-l31f krb5kdc[1872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:41:24.013301 2268 init.cc:377] Logged in from keytab as kudu/127.0.197.67@KRBTEST.COM (short username kudu)
I20260929 16:41:24.014987 2268 webserver.cc:460] Webserver started at http://127.0.197.67:44289/ using document root <none> and password file <none>
I20260929 16:41:24.015792 2268 fs_manager.cc:359] Metadata directory not provided
I20260929 16:41:24.015954 2268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:41:24.016352 2268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:41:24.019692 2268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/ts-2/data/instance:
uuid: "5c2a05b56511413a8bb0d0ce168d689e"
format_stamp: "Formatted at 2026-09-29 16:41:24 on dist-test-slave-l31f"
server_key: "78e985409c8622c1652201dd57e09646"
server_key_iv: "1dff2f40d7c68eba5262f16a22310309"
server_key_version: "encryptionkey@0"
I20260929 16:41:24.020594 2268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/ts-2/wal/instance:
uuid: "5c2a05b56511413a8bb0d0ce168d689e"
format_stamp: "Formatted at 2026-09-29 16:41:24 on dist-test-slave-l31f"
server_key: "78e985409c8622c1652201dd57e09646"
server_key_iv: "1dff2f40d7c68eba5262f16a22310309"
server_key_version: "encryptionkey@0"
I20260929 16:41:24.026414 2268 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 16:41:24.030470 2283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:24.032192 2268 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:41:24.032400 2268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/ts-2/wal
uuid: "5c2a05b56511413a8bb0d0ce168d689e"
format_stamp: "Formatted at 2026-09-29 16:41:24 on dist-test-slave-l31f"
server_key: "78e985409c8622c1652201dd57e09646"
server_key_iv: "1dff2f40d7c68eba5262f16a22310309"
server_key_version: "encryptionkey@0"
I20260929 16:41:24.032663 2268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:41:24.057834 2268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:41:24.064489 2268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:41:24.064807 2268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:41:24.067956 2268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:41:24.068288 2268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:24.068529 2268 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:41:24.068639 2268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:41:24 dist-test-slave-l31f krb5kdc[1872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.67@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:41:24.094805 2292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:24.068913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:45565 (local address 127.0.197.126:37717)
0929 16:41:24.069477 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:24.069484 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:24.070593 (+ 1109us) server_negotiation.cc:407] Connection header received
0929 16:41:24.072976 (+ 2383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:24.072985 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:24.073097 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:24.073265 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:24.075875 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:24.077001 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:24.079746 (+ 2745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:24.080501 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:24.080676 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:24.087079 (+ 6403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:24.087090 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:24.087098 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:24.087160 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:24.091262 (+ 4102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:24.092176 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:24.092182 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:24.092185 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:24.092334 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:24.092849 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:24.092858 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:24.092860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:24.093143 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:24.093309 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:24.094133 (+ 824us) server_negotiation.cc:299] Negotiation successful
0929 16:41:24.094374 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":287,"threads_started":1}
I20260929 16:41:24.097105 2293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:24.069397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37717 (local address 127.0.197.67:45565)
0929 16:41:24.070379 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:24.070428 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:24.072544 (+ 2116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:24.073639 (+ 1095us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:24.073651 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:24.074479 (+ 828us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:41:24.075519 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:24.075554 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:24.077199 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:24.077206 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:24.079508 (+ 2302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:24.079538 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:24.080847 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:24.080857 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:24.081278 (+ 421us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:24.082313 (+ 1035us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:41:24.082346 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:41:24.086537 (+ 4191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:41:24.091463 (+ 4926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:24.091475 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:24.091497 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:24.092013 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:24.092464 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:24.092470 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:24.092473 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:24.092704 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:24.093300 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:41:24.093310 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:41:24.093741 (+ 431us) client_negotiation.cc:769] Sending connection context
0929 16:41:24.094043 (+ 302us) client_negotiation.cc:240] Negotiation successful
0929 16:41:24.094656 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":394,"threads_started":1}
I20260929 16:41:24.122172 2375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:24.106292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37717 (local address 127.0.197.67:57473)
0929 16:41:24.107057 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:24.107080 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:24.107277 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:24.107648 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:24.107652 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:24.108168 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:41:24.108462 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:24.108473 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:24.109546 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:24.109551 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:24.110615 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:24.110629 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:24.111526 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:24.111532 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:24.111850 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:24.112792 (+ 942us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:41:24.112828 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:41:24.113848 (+ 1020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:41:24.118508 (+ 4660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:24.118517 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:24.118525 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:24.119430 (+ 905us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:24.120018 (+ 588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:24.120024 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:24.120027 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:24.120241 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:24.120939 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:41:24.120942 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:41:24.121119 (+ 177us) client_negotiation.cc:769] Sending connection context
0929 16:41:24.121245 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 16:41:24.121630 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":385,"threads_started":1}
I20260929 16:41:24.122202 2292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:24.106211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:57473 (local address 127.0.197.126:37717)
0929 16:41:24.106426 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:24.106431 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:24.107187 (+ 756us) server_negotiation.cc:407] Connection header received
0929 16:41:24.107437 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:24.107441 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:24.107510 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:24.107619 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:24.108623 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:24.109385 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:24.110784 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:24.111285 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:24.111440 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:24.114096 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:24.114102 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:24.114105 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:24.114142 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:24.118209 (+ 4067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:24.119623 (+ 1414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:24.119630 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:24.119634 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:24.119846 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:24.120405 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:24.120413 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:24.120419 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:24.120803 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:24.120984 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:24.121392 (+ 408us) server_negotiation.cc:299] Negotiation successful
0929 16:41:24.121690 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 16:41:24.122997 2268 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.67:33555
I20260929 16:41:24.123049 2403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.67:33555 every 8 connection(s)
I20260929 16:41:24.124607 2268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
I20260929 16:41:24.124905 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 2268
I20260929 16:41:24.125139 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/ts-2/wal/instance
I20260929 16:41:24.125701 789 external_mini_cluster.cc:1428] Setting key 52c3af6ab6ac08eb4f082bf77dcabc6c
I20260929 16:41:24.128242 2404 heartbeater.cc:347] Connected to a master server at 127.0.197.126:37717
I20260929 16:41:24.128523 2404 heartbeater.cc:464] Registering TS with master...
I20260929 16:41:24.129740 2404 heartbeater.cc:511] Master 127.0.197.126:37717 requested a full tablet report, sending...
I20260929 16:41:24.131654 1917 ts_manager.cc:194] Registered new tserver with Master: 5c2a05b56511413a8bb0d0ce168d689e (127.0.197.67:33555)
I20260929 16:41:24.132541 1917 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.67@KRBTEST.COM'} at 127.0.197.67:45565
I20260929 16:41:24.142861 789 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 16:41:24 dist-test-slave-l31f krb5kdc[1872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700084, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 16:41:24 dist-test-slave-l31f krb5kdc[1872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700084, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:41:24.196332 2292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:24.179310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44466 (local address 127.0.197.126:37717)
0929 16:41:24.179766 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:24.179773 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:24.180027 (+ 254us) server_negotiation.cc:407] Connection header received
0929 16:41:24.180178 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:24.180182 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:24.180257 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:24.180356 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:24.182046 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:24.182994 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:24.184827 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:24.185584 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:24.185733 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:24.190256 (+ 4523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:24.190262 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:24.190266 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:24.190312 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:24.193271 (+ 2959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:24.194117 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:24.194122 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:24.194124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:24.194284 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:24.194739 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:24.194743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:24.194745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:24.195187 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:24.195408 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:24.195840 (+ 432us) server_negotiation.cc:299] Negotiation successful
0929 16:41:24.196057 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264}
I20260929 16:41:24.201996 1917 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44466:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:41:24.209316 1917 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:41:24.238440 2417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:24.223137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.66:41639 (local address 127.0.0.1:45364)
0929 16:41:24.224061 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:24.224105 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:24.224461 (+ 356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:24.225893 (+ 1432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:24.225905 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:24.225948 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:41:24.226503 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:24.226537 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:24.229095 (+ 2558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:24.229104 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:24.232994 (+ 3890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:24.233032 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:24.237255 (+ 4223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:24.237263 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:24.237502 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:24.237572 (+ 70us) client_negotiation.cc:769] Sending connection context
0929 16:41:24.237708 (+ 136us) client_negotiation.cc:240] Negotiation successful
0929 16:41:24.238013 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":434,"threads_started":1}
I20260929 16:41:24.238581 2418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:24.223328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45364 (local address 127.0.197.66:41639)
0929 16:41:24.223881 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:24.223891 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:24.224333 (+ 442us) server_negotiation.cc:407] Connection header received
0929 16:41:24.224776 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:24.224789 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:24.225194 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:24.225585 (+ 391us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:41:24.226760 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:24.228861 (+ 2101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:24.233361 (+ 4500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:24.236984 (+ 3623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:24.237733 (+ 749us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:24.237998 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:24.238085 (+ 87us) server_negotiation.cc:299] Negotiation successful
0929 16:41:24.238291 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":863,"threads_started":1}
I20260929 16:41:24.243721 2194 tablet_service.cc:1467] Processing CreateTablet for tablet 0dbda514ee2a4fc5a74a8b945a115d15 (DEFAULT_TABLE table=test-table [id=bfbc670f4b264987b23c3b260e6f2712]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:41:24.245429 2194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dbda514ee2a4fc5a74a8b945a115d15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:41:24.247475 2422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:24.225229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50466 (local address 127.0.197.67:33555)
0929 16:41:24.239660 (+ 14431us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:24.239670 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:24.239715 (+ 45us) server_negotiation.cc:407] Connection header received
0929 16:41:24.239862 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:24.239871 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:24.240224 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:24.240441 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:41:24.241255 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:24.243043 (+ 1788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:24.244966 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:24.246390 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:24.246574 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:24.246846 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:24.246956 (+ 110us) server_negotiation.cc:299] Negotiation successful
0929 16:41:24.247138 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14229,"thread_start_us":14030,"threads_started":1}
I20260929 16:41:24.248068 2417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:24.224535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.67:33555 (local address 127.0.0.1:50466)
0929 16:41:24.239155 (+ 14620us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:24.239177 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:24.239286 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:24.240779 (+ 1493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:24.240784 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:24.240810 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:41:24.241035 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:24.241046 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:24.243268 (+ 2222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:24.243275 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:24.244711 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:24.244755 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:24.246587 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:24.246592 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:24.246775 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:24.246818 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 16:41:24.246917 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 16:41:24.247118 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14498,"mutex_wait_us":24,"thread_start_us":15040,"threads_started":1}
I20260929 16:41:24.248572 2419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:24.223968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:38669 (local address 127.0.0.1:58982)
0929 16:41:24.236594 (+ 12626us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:24.236622 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:24.238820 (+ 2198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:24.240590 (+ 1770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:24.240597 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:24.240625 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:41:24.241043 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:24.241052 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:24.243201 (+ 2149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:24.243206 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:24.245753 (+ 2547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:24.245770 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:24.247755 (+ 1985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:24.247761 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:24.247937 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:24.247974 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 16:41:24.248044 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 16:41:24.248180 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12466,"mutex_wait_us":26,"thread_start_us":12293,"threads_started":1}
I20260929 16:41:24.249256 2420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:24.224279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58982 (local address 127.0.197.65:38669)
0929 16:41:24.240051 (+ 15772us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:24.240061 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:24.240086 (+ 25us) server_negotiation.cc:407] Connection header received
0929 16:41:24.240179 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:24.240187 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:24.240382 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:24.241247 (+ 865us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:41:24.241460 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:24.243043 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:24.245976 (+ 2933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:24.247517 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:24.248516 (+ 999us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:24.248737 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:24.248808 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 16:41:24.248977 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15621,"thread_start_us":15501,"threads_started":1}
I20260929 16:41:24.252705 2054 tablet_service.cc:1467] Processing CreateTablet for tablet 0dbda514ee2a4fc5a74a8b945a115d15 (DEFAULT_TABLE table=test-table [id=bfbc670f4b264987b23c3b260e6f2712]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:41:24.252705 2334 tablet_service.cc:1467] Processing CreateTablet for tablet 0dbda514ee2a4fc5a74a8b945a115d15 (DEFAULT_TABLE table=test-table [id=bfbc670f4b264987b23c3b260e6f2712]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:41:24.254256 2054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dbda514ee2a4fc5a74a8b945a115d15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:41:24.254268 2334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dbda514ee2a4fc5a74a8b945a115d15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:41:24.259451 2423 tablet_bootstrap.cc:492] T 0dbda514ee2a4fc5a74a8b945a115d15 P 46bb3be715984c2aa30664a8fb35cf7b: Bootstrap starting.
I20260929 16:41:24.263067 2423 tablet_bootstrap.cc:654] T 0dbda514ee2a4fc5a74a8b945a115d15 P 46bb3be715984c2aa30664a8fb35cf7b: Neither blocks nor log segments found. Creating new log.
I20260929 16:41:24.264513 2423 log.cc:824] T 0dbda514ee2a4fc5a74a8b945a115d15 P 46bb3be715984c2aa30664a8fb35cf7b: Log is configured to *not* fsync() on all Append() calls
I20260929 16:41:24.267798 2423 tablet_bootstrap.cc:492] T 0dbda514ee2a4fc5a74a8b945a115d15 P 46bb3be715984c2aa30664a8fb35cf7b: No bootstrap required, opened a new log
I20260929 16:41:24.268141 2423 ts_tablet_manager.cc:1397] T 0dbda514ee2a4fc5a74a8b945a115d15 P 46bb3be715984c2aa30664a8fb35cf7b: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260929 16:41:24.270167 2426 tablet_bootstrap.cc:492] T 0dbda514ee2a4fc5a74a8b945a115d15 P 5c2a05b56511413a8bb0d0ce168d689e: Bootstrap starting.
I20260929 16:41:24.270978 2425 tablet_bootstrap.cc:492] T 0dbda514ee2a4fc5a74a8b945a115d15 P f0a80dcdaf6d4a299f3aefbd155e3783: Bootstrap starting.
I20260929 16:41:24.275194 2425 tablet_bootstrap.cc:654] T 0dbda514ee2a4fc5a74a8b945a115d15 P f0a80dcdaf6d4a299f3aefbd155e3783: Neither blocks nor log segments found. Creating new log.
I20260929 16:41:24.276628 2426 tablet_bootstrap.cc:654] T 0dbda514ee2a4fc5a74a8b945a115d15 P 5c2a05b56511413a8bb0d0ce168d689e: Neither blocks nor log segments found. Creating new log.
I20260929 16:41:24.276611 2425 log.cc:824] T 0dbda514ee2a4fc5a74a8b945a115d15 P f0a80dcdaf6d4a299f3aefbd155e3783: Log is configured to *not* fsync() on all Append() calls
I20260929 16:41:24.278404 2426 log.cc:824] T 0dbda514ee2a4fc5a74a8b945a115d15 P 5c2a05b56511413a8bb0d0ce168d689e: Log is configured to *not* fsync() on all Append() calls
I20260929 16:41:24.278838 2423 raft_consensus.cc:348] T 0dbda514ee2a4fc5a74a8b945a115d15 P 46bb3be715984c2aa30664a8fb35cf7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46bb3be715984c2aa30664a8fb35cf7b" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 41639 } } peers { permanent_uuid: "5c2a05b56511413a8bb0d0ce168d689e" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33555 } } peers { permanent_uuid: "f0a80dcdaf6d4a299f3aefbd155e3783" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 38669 } }
I20260929 16:41:24.279361 2423 raft_consensus.cc:374] T 0dbda514ee2a4fc5a74a8b945a115d15 P 46bb3be715984c2aa30664a8fb35cf7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:41:24.279485 2423 raft_consensus.cc:729] T 0dbda514ee2a4fc5a74a8b945a115d15 P 46bb3be715984c2aa30664a8fb35cf7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46bb3be715984c2aa30664a8fb35cf7b, State: Initialized, Role: FOLLOWER
I20260929 16:41:24.280212 2423 consensus_queue.cc:260] T 0dbda514ee2a4fc5a74a8b945a115d15 P 46bb3be715984c2aa30664a8fb35cf7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46bb3be715984c2aa30664a8fb35cf7b" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 41639 } } peers { permanent_uuid: "5c2a05b56511413a8bb0d0ce168d689e" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33555 } } peers { permanent_uuid: "f0a80dcdaf6d4a299f3aefbd155e3783" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 38669 } }
I20260929 16:41:24.280683 2425 tablet_bootstrap.cc:492] T 0dbda514ee2a4fc5a74a8b945a115d15 P f0a80dcdaf6d4a299f3aefbd155e3783: No bootstrap required, opened a new log
I20260929 16:41:24.281038 2425 ts_tablet_manager.cc:1397] T 0dbda514ee2a4fc5a74a8b945a115d15 P f0a80dcdaf6d4a299f3aefbd155e3783: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.007s
I20260929 16:41:24.281513 2264 heartbeater.cc:503] Master 127.0.197.126:37717 was elected leader, sending a full tablet report...
I20260929 16:41:24.281656 2426 tablet_bootstrap.cc:492] T 0dbda514ee2a4fc5a74a8b945a115d15 P 5c2a05b56511413a8bb0d0ce168d689e: No bootstrap required, opened a new log
I20260929 16:41:24.281936 2426 ts_tablet_manager.cc:1397] T 0dbda514ee2a4fc5a74a8b945a115d15 P 5c2a05b56511413a8bb0d0ce168d689e: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.003s
I20260929 16:41:24.282176 2423 ts_tablet_manager.cc:1428] T 0dbda514ee2a4fc5a74a8b945a115d15 P 46bb3be715984c2aa30664a8fb35cf7b: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
I20260929 16:41:24.292204 2425 raft_consensus.cc:348] T 0dbda514ee2a4fc5a74a8b945a115d15 P f0a80dcdaf6d4a299f3aefbd155e3783 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46bb3be715984c2aa30664a8fb35cf7b" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 41639 } } peers { permanent_uuid: "5c2a05b56511413a8bb0d0ce168d689e" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33555 } } peers { permanent_uuid: "f0a80dcdaf6d4a299f3aefbd155e3783" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 38669 } }
I20260929 16:41:24.292204 2426 raft_consensus.cc:348] T 0dbda514ee2a4fc5a74a8b945a115d15 P 5c2a05b56511413a8bb0d0ce168d689e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46bb3be715984c2aa30664a8fb35cf7b" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 41639 } } peers { permanent_uuid: "5c2a05b56511413a8bb0d0ce168d689e" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33555 } } peers { permanent_uuid: "f0a80dcdaf6d4a299f3aefbd155e3783" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 38669 } }
I20260929 16:41:24.292625 2425 raft_consensus.cc:374] T 0dbda514ee2a4fc5a74a8b945a115d15 P f0a80dcdaf6d4a299f3aefbd155e3783 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:41:24.292665 2426 raft_consensus.cc:374] T 0dbda514ee2a4fc5a74a8b945a115d15 P 5c2a05b56511413a8bb0d0ce168d689e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:41:24.292765 2425 raft_consensus.cc:729] T 0dbda514ee2a4fc5a74a8b945a115d15 P f0a80dcdaf6d4a299f3aefbd155e3783 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0a80dcdaf6d4a299f3aefbd155e3783, State: Initialized, Role: FOLLOWER
I20260929 16:41:24.292784 2426 raft_consensus.cc:729] T 0dbda514ee2a4fc5a74a8b945a115d15 P 5c2a05b56511413a8bb0d0ce168d689e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c2a05b56511413a8bb0d0ce168d689e, State: Initialized, Role: FOLLOWER
I20260929 16:41:24.293470 2426 consensus_queue.cc:260] T 0dbda514ee2a4fc5a74a8b945a115d15 P 5c2a05b56511413a8bb0d0ce168d689e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46bb3be715984c2aa30664a8fb35cf7b" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 41639 } } peers { permanent_uuid: "5c2a05b56511413a8bb0d0ce168d689e" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33555 } } peers { permanent_uuid: "f0a80dcdaf6d4a299f3aefbd155e3783" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 38669 } }
I20260929 16:41:24.293468 2425 consensus_queue.cc:260] T 0dbda514ee2a4fc5a74a8b945a115d15 P f0a80dcdaf6d4a299f3aefbd155e3783 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46bb3be715984c2aa30664a8fb35cf7b" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 41639 } } peers { permanent_uuid: "5c2a05b56511413a8bb0d0ce168d689e" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33555 } } peers { permanent_uuid: "f0a80dcdaf6d4a299f3aefbd155e3783" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 38669 } }
I20260929 16:41:24.294680 2404 heartbeater.cc:503] Master 127.0.197.126:37717 was elected leader, sending a full tablet report...
I20260929 16:41:24.295462 2425 ts_tablet_manager.cc:1428] T 0dbda514ee2a4fc5a74a8b945a115d15 P f0a80dcdaf6d4a299f3aefbd155e3783: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260929 16:41:24.295483 2426 ts_tablet_manager.cc:1428] T 0dbda514ee2a4fc5a74a8b945a115d15 P 5c2a05b56511413a8bb0d0ce168d689e: Time spent starting tablet: real 0.013s user 0.010s sys 0.005s
W20260929 16:41:24.302266 2265 tablet.cc:2369] T 0dbda514ee2a4fc5a74a8b945a115d15 P 46bb3be715984c2aa30664a8fb35cf7b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:41:24.381778 2405 tablet.cc:2369] T 0dbda514ee2a4fc5a74a8b945a115d15 P 5c2a05b56511413a8bb0d0ce168d689e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:41:24.482402 2125 tablet.cc:2369] T 0dbda514ee2a4fc5a74a8b945a115d15 P f0a80dcdaf6d4a299f3aefbd155e3783: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:41:24.538026 2429 raft_consensus.cc:482] T 0dbda514ee2a4fc5a74a8b945a115d15 P 46bb3be715984c2aa30664a8fb35cf7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:41:24.538350 2429 raft_consensus.cc:504] T 0dbda514ee2a4fc5a74a8b945a115d15 P 46bb3be715984c2aa30664a8fb35cf7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46bb3be715984c2aa30664a8fb35cf7b" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 41639 } } peers { permanent_uuid: "5c2a05b56511413a8bb0d0ce168d689e" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33555 } } peers { permanent_uuid: "f0a80dcdaf6d4a299f3aefbd155e3783" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 38669 } }
I20260929 16:41:24.540103 2429 leader_election.cc:288] T 0dbda514ee2a4fc5a74a8b945a115d15 P 46bb3be715984c2aa30664a8fb35cf7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c2a05b56511413a8bb0d0ce168d689e (127.0.197.67:33555), f0a80dcdaf6d4a299f3aefbd155e3783 (127.0.197.65:38669)
I20260929 16:41:24.548183 2432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:24.540401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.67:33555 (local address 127.0.197.66:37591)
0929 16:41:24.541607 (+ 1206us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:24.541624 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:24.541761 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:24.542118 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:24.542122 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:24.542155 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:41:24.542523 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:24.542533 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:24.543686 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:24.543691 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:24.545933 (+ 2242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:24.545945 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:24.547427 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:24.547432 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:24.547575 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:24.547626 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 16:41:24.547744 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 16:41:24.547864 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1072,"mutex_wait_us":113,"thread_start_us":850,"threads_started":1}
I20260929 16:41:24.548177 2422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:24.540716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:37591 (local address 127.0.197.67:33555)
0929 16:41:24.540931 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:24.540938 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:24.541733 (+ 795us) server_negotiation.cc:407] Connection header received
0929 16:41:24.541899 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:24.541906 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:24.541987 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:24.542094 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:41:24.542679 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:24.543555 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:24.546125 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:24.547106 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:24.547298 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:24.547369 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:24.547794 (+ 425us) server_negotiation.cc:299] Negotiation successful
0929 16:41:24.547939 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 16:41:24.549507 2420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:24.540747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:46917 (local address 127.0.197.65:38669)
0929 16:41:24.541035 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:24.541046 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:24.541913 (+ 867us) server_negotiation.cc:407] Connection header received
0929 16:41:24.542103 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:24.542108 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:24.542201 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:24.542306 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:41:24.542937 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:24.543758 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:24.546057 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:24.547050 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:24.547232 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:24.547334 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:24.549103 (+ 1769us) server_negotiation.cc:299] Negotiation successful
0929 16:41:24.549258 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260929 16:41:24.549355 2354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dbda514ee2a4fc5a74a8b945a115d15" candidate_uuid: "46bb3be715984c2aa30664a8fb35cf7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c2a05b56511413a8bb0d0ce168d689e" is_pre_election: true
I20260929 16:41:24.549621 2433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:24.540640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:38669 (local address 127.0.197.66:46917)
0929 16:41:24.541760 (+ 1120us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:24.541778 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:24.541937 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:24.542342 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:24.542345 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:24.542364 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:41:24.542762 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:24.542775 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:24.543905 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:24.543912 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:24.545876 (+ 1964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:24.545891 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:24.548707 (+ 2816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:24.548714 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:24.548906 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:24.548941 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 16:41:24.549046 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 16:41:24.549188 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":967,"mutex_wait_us":8,"thread_start_us":645,"threads_started":1}
I20260929 16:41:24.549898 2354 raft_consensus.cc:2454] T 0dbda514ee2a4fc5a74a8b945a115d15 P 5c2a05b56511413a8bb0d0ce168d689e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46bb3be715984c2aa30664a8fb35cf7b in term 0.
I20260929 16:41:24.550624 2146 leader_election.cc:302] T 0dbda514ee2a4fc5a74a8b945a115d15 P 46bb3be715984c2aa30664a8fb35cf7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46bb3be715984c2aa30664a8fb35cf7b, 5c2a05b56511413a8bb0d0ce168d689e; no voters:
I20260929 16:41:24.550715 2078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dbda514ee2a4fc5a74a8b945a115d15" candidate_uuid: "46bb3be715984c2aa30664a8fb35cf7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0a80dcdaf6d4a299f3aefbd155e3783" is_pre_election: true
I20260929 16:41:24.551168 2429 raft_consensus.cc:2781] T 0dbda514ee2a4fc5a74a8b945a115d15 P 46bb3be715984c2aa30664a8fb35cf7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:41:24.551340 2429 raft_consensus.cc:482] T 0dbda514ee2a4fc5a74a8b945a115d15 P 46bb3be715984c2aa30664a8fb35cf7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:41:24.551291 2078 raft_consensus.cc:2454] T 0dbda514ee2a4fc5a74a8b945a115d15 P f0a80dcdaf6d4a299f3aefbd155e3783 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46bb3be715984c2aa30664a8fb35cf7b in term 0.
I20260929 16:41:24.551474 2429 raft_consensus.cc:3037] T 0dbda514ee2a4fc5a74a8b945a115d15 P 46bb3be715984c2aa30664a8fb35cf7b [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:41:24.554927 2429 raft_consensus.cc:504] T 0dbda514ee2a4fc5a74a8b945a115d15 P 46bb3be715984c2aa30664a8fb35cf7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46bb3be715984c2aa30664a8fb35cf7b" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 41639 } } peers { permanent_uuid: "5c2a05b56511413a8bb0d0ce168d689e" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33555 } } peers { permanent_uuid: "f0a80dcdaf6d4a299f3aefbd155e3783" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 38669 } }
I20260929 16:41:24.555931 2429 leader_election.cc:288] T 0dbda514ee2a4fc5a74a8b945a115d15 P 46bb3be715984c2aa30664a8fb35cf7b [CANDIDATE]: Term 1 election: Requested vote from peers 5c2a05b56511413a8bb0d0ce168d689e (127.0.197.67:33555), f0a80dcdaf6d4a299f3aefbd155e3783 (127.0.197.65:38669)
I20260929 16:41:24.556532 2078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dbda514ee2a4fc5a74a8b945a115d15" candidate_uuid: "46bb3be715984c2aa30664a8fb35cf7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0a80dcdaf6d4a299f3aefbd155e3783"
I20260929 16:41:24.556583 2354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dbda514ee2a4fc5a74a8b945a115d15" candidate_uuid: "46bb3be715984c2aa30664a8fb35cf7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c2a05b56511413a8bb0d0ce168d689e"
I20260929 16:41:24.556779 2078 raft_consensus.cc:3037] T 0dbda514ee2a4fc5a74a8b945a115d15 P f0a80dcdaf6d4a299f3aefbd155e3783 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:41:24.556779 2354 raft_consensus.cc:3037] T 0dbda514ee2a4fc5a74a8b945a115d15 P 5c2a05b56511413a8bb0d0ce168d689e [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:41:24.560617 2078 raft_consensus.cc:2454] T 0dbda514ee2a4fc5a74a8b945a115d15 P f0a80dcdaf6d4a299f3aefbd155e3783 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46bb3be715984c2aa30664a8fb35cf7b in term 1.
I20260929 16:41:24.560632 2354 raft_consensus.cc:2454] T 0dbda514ee2a4fc5a74a8b945a115d15 P 5c2a05b56511413a8bb0d0ce168d689e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46bb3be715984c2aa30664a8fb35cf7b in term 1.
I20260929 16:41:24.561194 2147 leader_election.cc:302] T 0dbda514ee2a4fc5a74a8b945a115d15 P 46bb3be715984c2aa30664a8fb35cf7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46bb3be715984c2aa30664a8fb35cf7b, f0a80dcdaf6d4a299f3aefbd155e3783; no voters:
I20260929 16:41:24.561599 2429 raft_consensus.cc:2781] T 0dbda514ee2a4fc5a74a8b945a115d15 P 46bb3be715984c2aa30664a8fb35cf7b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:41:24.562201 2429 raft_consensus.cc:686] T 0dbda514ee2a4fc5a74a8b945a115d15 P 46bb3be715984c2aa30664a8fb35cf7b [term 1 LEADER]: Becoming Leader. State: Replica: 46bb3be715984c2aa30664a8fb35cf7b, State: Running, Role: LEADER
I20260929 16:41:24.563228 2429 consensus_queue.cc:237] T 0dbda514ee2a4fc5a74a8b945a115d15 P 46bb3be715984c2aa30664a8fb35cf7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46bb3be715984c2aa30664a8fb35cf7b" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 41639 } } peers { permanent_uuid: "5c2a05b56511413a8bb0d0ce168d689e" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33555 } } peers { permanent_uuid: "f0a80dcdaf6d4a299f3aefbd155e3783" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 38669 } }
I20260929 16:41:24.567907 1917 catalog_manager.cc:5495] T 0dbda514ee2a4fc5a74a8b945a115d15 P 46bb3be715984c2aa30664a8fb35cf7b reported cstate change: term changed from 0 to 1, leader changed from <none> to 46bb3be715984c2aa30664a8fb35cf7b (127.0.197.66). New cstate: current_term: 1 leader_uuid: "46bb3be715984c2aa30664a8fb35cf7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46bb3be715984c2aa30664a8fb35cf7b" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c2a05b56511413a8bb0d0ce168d689e" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0a80dcdaf6d4a299f3aefbd155e3783" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 38669 } health_report { overall_health: UNKNOWN } } }
I20260929 16:41:24.624516 2418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:24.618604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45372 (local address 127.0.197.66:41639)
0929 16:41:24.618828 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:24.618836 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:24.618862 (+ 26us) server_negotiation.cc:407] Connection header received
0929 16:41:24.619192 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:24.619196 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:24.619277 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:24.619432 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:41:24.619799 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:24.620528 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:24.621888 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:24.622301 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:24.622443 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:24.623056 (+ 613us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:41:24.623775 (+ 719us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:41:24.623996 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:24.624134 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 16:41:24.624229 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 16:41:24.634189 2078 raft_consensus.cc:1261] T 0dbda514ee2a4fc5a74a8b945a115d15 P f0a80dcdaf6d4a299f3aefbd155e3783 [term 1 FOLLOWER]: Refusing update from remote peer 46bb3be715984c2aa30664a8fb35cf7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:41:24.634189 2354 raft_consensus.cc:1261] T 0dbda514ee2a4fc5a74a8b945a115d15 P 5c2a05b56511413a8bb0d0ce168d689e [term 1 FOLLOWER]: Refusing update from remote peer 46bb3be715984c2aa30664a8fb35cf7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:41:24.635347 2434 consensus_queue.cc:1035] T 0dbda514ee2a4fc5a74a8b945a115d15 P 46bb3be715984c2aa30664a8fb35cf7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c2a05b56511413a8bb0d0ce168d689e" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:41:24.635720 2429 consensus_queue.cc:1035] T 0dbda514ee2a4fc5a74a8b945a115d15 P 46bb3be715984c2aa30664a8fb35cf7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a80dcdaf6d4a299f3aefbd155e3783" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 38669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:41:24.672499 1916 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:41:24.672825 1916 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:41:24.677211 1916 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b1e9d1fcf7c2470883bd4ca247fff193: Sending DeleteTablet for 3 replicas of tablet 0dbda514ee2a4fc5a74a8b945a115d15
I20260929 16:41:24.678622 2194 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dbda514ee2a4fc5a74a8b945a115d15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:41:24 UTC) from {username='kudu', principal='kudu/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:45364
I20260929 16:41:24.678593 2334 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dbda514ee2a4fc5a74a8b945a115d15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:41:24 UTC) from {username='kudu', principal='kudu/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:50466
I20260929 16:41:24.679459 2054 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dbda514ee2a4fc5a74a8b945a115d15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:41:24 UTC) from {username='kudu', principal='kudu/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:58982
I20260929 16:41:24.680564 1916 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44466:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:41:24.681468 1916 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:41:24.692236 2448 tablet_replica.cc:330] T 0dbda514ee2a4fc5a74a8b945a115d15 P f0a80dcdaf6d4a299f3aefbd155e3783: stopping tablet replica
I20260929 16:41:24.692657 2193 tablet_service.cc:1467] Processing CreateTablet for tablet 3d64154ac1ba448d876f47f50221d313 (DEFAULT_TABLE table=test-table [id=59738d2a417a45b8bb435121ce8cada9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:41:24.692652 2053 tablet_service.cc:1467] Processing CreateTablet for tablet 3d64154ac1ba448d876f47f50221d313 (DEFAULT_TABLE table=test-table [id=59738d2a417a45b8bb435121ce8cada9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:41:24.693368 2193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d64154ac1ba448d876f47f50221d313. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:41:24.693709 2053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d64154ac1ba448d876f47f50221d313. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:41:24.694044 2448 raft_consensus.cc:2229] T 0dbda514ee2a4fc5a74a8b945a115d15 P f0a80dcdaf6d4a299f3aefbd155e3783 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:41:24.694952 2448 raft_consensus.cc:2258] T 0dbda514ee2a4fc5a74a8b945a115d15 P f0a80dcdaf6d4a299f3aefbd155e3783 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:41:24.700768 2333 tablet_service.cc:1467] Processing CreateTablet for tablet 3d64154ac1ba448d876f47f50221d313 (DEFAULT_TABLE table=test-table [id=59738d2a417a45b8bb435121ce8cada9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:41:24.701781 2333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d64154ac1ba448d876f47f50221d313. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:41:24.702018 2423 tablet_bootstrap.cc:492] T 3d64154ac1ba448d876f47f50221d313 P 46bb3be715984c2aa30664a8fb35cf7b: Bootstrap starting.
I20260929 16:41:24.703888 2423 tablet_bootstrap.cc:654] T 3d64154ac1ba448d876f47f50221d313 P 46bb3be715984c2aa30664a8fb35cf7b: Neither blocks nor log segments found. Creating new log.
I20260929 16:41:24.703897 2448 ts_tablet_manager.cc:1905] T 0dbda514ee2a4fc5a74a8b945a115d15 P f0a80dcdaf6d4a299f3aefbd155e3783: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:41:24.711787 2447 tablet_replica.cc:330] T 0dbda514ee2a4fc5a74a8b945a115d15 P 46bb3be715984c2aa30664a8fb35cf7b: stopping tablet replica
I20260929 16:41:24.712471 2447 raft_consensus.cc:2229] T 0dbda514ee2a4fc5a74a8b945a115d15 P 46bb3be715984c2aa30664a8fb35cf7b [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:41:24.713440 2447 raft_consensus.cc:2258] T 0dbda514ee2a4fc5a74a8b945a115d15 P 46bb3be715984c2aa30664a8fb35cf7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:41:24.714893 2426 tablet_bootstrap.cc:492] T 3d64154ac1ba448d876f47f50221d313 P 5c2a05b56511413a8bb0d0ce168d689e: Bootstrap starting.
I20260929 16:41:24.716481 2447 ts_tablet_manager.cc:1905] T 0dbda514ee2a4fc5a74a8b945a115d15 P 46bb3be715984c2aa30664a8fb35cf7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:41:24.717914 2426 tablet_bootstrap.cc:654] T 3d64154ac1ba448d876f47f50221d313 P 5c2a05b56511413a8bb0d0ce168d689e: Neither blocks nor log segments found. Creating new log.
I20260929 16:41:24.718518 2446 tablet_replica.cc:330] T 0dbda514ee2a4fc5a74a8b945a115d15 P 5c2a05b56511413a8bb0d0ce168d689e: stopping tablet replica
I20260929 16:41:24.718937 2446 raft_consensus.cc:2229] T 0dbda514ee2a4fc5a74a8b945a115d15 P 5c2a05b56511413a8bb0d0ce168d689e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:41:24.719502 2446 raft_consensus.cc:2258] T 0dbda514ee2a4fc5a74a8b945a115d15 P 5c2a05b56511413a8bb0d0ce168d689e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:41:24.725754 2447 ts_tablet_manager.cc:1918] T 0dbda514ee2a4fc5a74a8b945a115d15 P 46bb3be715984c2aa30664a8fb35cf7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:41:24.726001 2447 log.cc:1196] T 0dbda514ee2a4fc5a74a8b945a115d15 P 46bb3be715984c2aa30664a8fb35cf7b: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/ts-1/wal/wals/0dbda514ee2a4fc5a74a8b945a115d15
I20260929 16:41:24.726820 2425 tablet_bootstrap.cc:492] T 3d64154ac1ba448d876f47f50221d313 P f0a80dcdaf6d4a299f3aefbd155e3783: Bootstrap starting.
I20260929 16:41:24.728668 2425 tablet_bootstrap.cc:654] T 3d64154ac1ba448d876f47f50221d313 P f0a80dcdaf6d4a299f3aefbd155e3783: Neither blocks nor log segments found. Creating new log.
I20260929 16:41:24.729730 2448 ts_tablet_manager.cc:1918] T 0dbda514ee2a4fc5a74a8b945a115d15 P f0a80dcdaf6d4a299f3aefbd155e3783: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:41:24.729928 2423 tablet_bootstrap.cc:492] T 3d64154ac1ba448d876f47f50221d313 P 46bb3be715984c2aa30664a8fb35cf7b: No bootstrap required, opened a new log
I20260929 16:41:24.730090 2448 log.cc:1196] T 0dbda514ee2a4fc5a74a8b945a115d15 P f0a80dcdaf6d4a299f3aefbd155e3783: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/ts-0/wal/wals/0dbda514ee2a4fc5a74a8b945a115d15
I20260929 16:41:24.730122 2423 ts_tablet_manager.cc:1397] T 3d64154ac1ba448d876f47f50221d313 P 46bb3be715984c2aa30664a8fb35cf7b: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.008s
I20260929 16:41:24.730747 2426 tablet_bootstrap.cc:492] T 3d64154ac1ba448d876f47f50221d313 P 5c2a05b56511413a8bb0d0ce168d689e: No bootstrap required, opened a new log
I20260929 16:41:24.730918 2426 ts_tablet_manager.cc:1397] T 3d64154ac1ba448d876f47f50221d313 P 5c2a05b56511413a8bb0d0ce168d689e: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.003s
I20260929 16:41:24.731057 2448 ts_tablet_manager.cc:1939] T 0dbda514ee2a4fc5a74a8b945a115d15 P f0a80dcdaf6d4a299f3aefbd155e3783: Deleting consensus metadata
I20260929 16:41:24.730975 2423 raft_consensus.cc:348] T 3d64154ac1ba448d876f47f50221d313 P 46bb3be715984c2aa30664a8fb35cf7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a80dcdaf6d4a299f3aefbd155e3783" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 38669 } } peers { permanent_uuid: "46bb3be715984c2aa30664a8fb35cf7b" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 41639 } } peers { permanent_uuid: "5c2a05b56511413a8bb0d0ce168d689e" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33555 } }
I20260929 16:41:24.731274 2423 raft_consensus.cc:374] T 3d64154ac1ba448d876f47f50221d313 P 46bb3be715984c2aa30664a8fb35cf7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:41:24.731376 2423 raft_consensus.cc:729] T 3d64154ac1ba448d876f47f50221d313 P 46bb3be715984c2aa30664a8fb35cf7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46bb3be715984c2aa30664a8fb35cf7b, State: Initialized, Role: FOLLOWER
I20260929 16:41:24.731640 2423 consensus_queue.cc:260] T 3d64154ac1ba448d876f47f50221d313 P 46bb3be715984c2aa30664a8fb35cf7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a80dcdaf6d4a299f3aefbd155e3783" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 38669 } } peers { permanent_uuid: "46bb3be715984c2aa30664a8fb35cf7b" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 41639 } } peers { permanent_uuid: "5c2a05b56511413a8bb0d0ce168d689e" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33555 } }
I20260929 16:41:24.733108 2446 ts_tablet_manager.cc:1905] T 0dbda514ee2a4fc5a74a8b945a115d15 P 5c2a05b56511413a8bb0d0ce168d689e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:41:24.733162 2423 ts_tablet_manager.cc:1428] T 3d64154ac1ba448d876f47f50221d313 P 46bb3be715984c2aa30664a8fb35cf7b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 16:41:24.733479 1904 catalog_manager.cc:4841] TS f0a80dcdaf6d4a299f3aefbd155e3783 (127.0.197.65:38669): tablet 0dbda514ee2a4fc5a74a8b945a115d15 (table test-table [id=bfbc670f4b264987b23c3b260e6f2712]) successfully deleted
I20260929 16:41:24.735489 2447 ts_tablet_manager.cc:1939] T 0dbda514ee2a4fc5a74a8b945a115d15 P 46bb3be715984c2aa30664a8fb35cf7b: Deleting consensus metadata
I20260929 16:41:24.738413 2426 raft_consensus.cc:348] T 3d64154ac1ba448d876f47f50221d313 P 5c2a05b56511413a8bb0d0ce168d689e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a80dcdaf6d4a299f3aefbd155e3783" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 38669 } } peers { permanent_uuid: "46bb3be715984c2aa30664a8fb35cf7b" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 41639 } } peers { permanent_uuid: "5c2a05b56511413a8bb0d0ce168d689e" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33555 } }
I20260929 16:41:24.738703 2426 raft_consensus.cc:374] T 3d64154ac1ba448d876f47f50221d313 P 5c2a05b56511413a8bb0d0ce168d689e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:41:24.738819 2426 raft_consensus.cc:729] T 3d64154ac1ba448d876f47f50221d313 P 5c2a05b56511413a8bb0d0ce168d689e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c2a05b56511413a8bb0d0ce168d689e, State: Initialized, Role: FOLLOWER
I20260929 16:41:24.739141 2426 consensus_queue.cc:260] T 3d64154ac1ba448d876f47f50221d313 P 5c2a05b56511413a8bb0d0ce168d689e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a80dcdaf6d4a299f3aefbd155e3783" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 38669 } } peers { permanent_uuid: "46bb3be715984c2aa30664a8fb35cf7b" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 41639 } } peers { permanent_uuid: "5c2a05b56511413a8bb0d0ce168d689e" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33555 } }
I20260929 16:41:24.740170 1903 catalog_manager.cc:4841] TS 46bb3be715984c2aa30664a8fb35cf7b (127.0.197.66:41639): tablet 0dbda514ee2a4fc5a74a8b945a115d15 (table test-table [id=bfbc670f4b264987b23c3b260e6f2712]) successfully deleted
I20260929 16:41:24.740535 2426 ts_tablet_manager.cc:1428] T 3d64154ac1ba448d876f47f50221d313 P 5c2a05b56511413a8bb0d0ce168d689e: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20260929 16:41:24.743062 2425 tablet_bootstrap.cc:492] T 3d64154ac1ba448d876f47f50221d313 P f0a80dcdaf6d4a299f3aefbd155e3783: No bootstrap required, opened a new log
I20260929 16:41:24.743287 2425 ts_tablet_manager.cc:1397] T 3d64154ac1ba448d876f47f50221d313 P f0a80dcdaf6d4a299f3aefbd155e3783: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.000s
I20260929 16:41:24.744319 2446 ts_tablet_manager.cc:1918] T 0dbda514ee2a4fc5a74a8b945a115d15 P 5c2a05b56511413a8bb0d0ce168d689e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:41:24.744233 2425 raft_consensus.cc:348] T 3d64154ac1ba448d876f47f50221d313 P f0a80dcdaf6d4a299f3aefbd155e3783 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a80dcdaf6d4a299f3aefbd155e3783" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 38669 } } peers { permanent_uuid: "46bb3be715984c2aa30664a8fb35cf7b" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 41639 } } peers { permanent_uuid: "5c2a05b56511413a8bb0d0ce168d689e" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33555 } }
I20260929 16:41:24.744498 2446 log.cc:1196] T 0dbda514ee2a4fc5a74a8b945a115d15 P 5c2a05b56511413a8bb0d0ce168d689e: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/ts-2/wal/wals/0dbda514ee2a4fc5a74a8b945a115d15
I20260929 16:41:24.744659 2425 raft_consensus.cc:374] T 3d64154ac1ba448d876f47f50221d313 P f0a80dcdaf6d4a299f3aefbd155e3783 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:41:24.744802 2425 raft_consensus.cc:729] T 3d64154ac1ba448d876f47f50221d313 P f0a80dcdaf6d4a299f3aefbd155e3783 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0a80dcdaf6d4a299f3aefbd155e3783, State: Initialized, Role: FOLLOWER
I20260929 16:41:24.744779 2431 raft_consensus.cc:482] T 3d64154ac1ba448d876f47f50221d313 P 5c2a05b56511413a8bb0d0ce168d689e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:41:24.745139 2446 ts_tablet_manager.cc:1939] T 0dbda514ee2a4fc5a74a8b945a115d15 P 5c2a05b56511413a8bb0d0ce168d689e: Deleting consensus metadata
I20260929 16:41:24.744973 2431 raft_consensus.cc:504] T 3d64154ac1ba448d876f47f50221d313 P 5c2a05b56511413a8bb0d0ce168d689e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a80dcdaf6d4a299f3aefbd155e3783" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 38669 } } peers { permanent_uuid: "46bb3be715984c2aa30664a8fb35cf7b" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 41639 } } peers { permanent_uuid: "5c2a05b56511413a8bb0d0ce168d689e" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33555 } }
I20260929 16:41:24.745410 2425 consensus_queue.cc:260] T 3d64154ac1ba448d876f47f50221d313 P f0a80dcdaf6d4a299f3aefbd155e3783 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a80dcdaf6d4a299f3aefbd155e3783" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 38669 } } peers { permanent_uuid: "46bb3be715984c2aa30664a8fb35cf7b" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 41639 } } peers { permanent_uuid: "5c2a05b56511413a8bb0d0ce168d689e" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33555 } }
I20260929 16:41:24.746184 2425 ts_tablet_manager.cc:1428] T 3d64154ac1ba448d876f47f50221d313 P f0a80dcdaf6d4a299f3aefbd155e3783: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 16:41:24.746937 2431 leader_election.cc:288] T 3d64154ac1ba448d876f47f50221d313 P 5c2a05b56511413a8bb0d0ce168d689e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0a80dcdaf6d4a299f3aefbd155e3783 (127.0.197.65:38669), 46bb3be715984c2aa30664a8fb35cf7b (127.0.197.66:41639)
I20260929 16:41:24.751593 1903 catalog_manager.cc:4841] TS 5c2a05b56511413a8bb0d0ce168d689e (127.0.197.67:33555): tablet 0dbda514ee2a4fc5a74a8b945a115d15 (table test-table [id=bfbc670f4b264987b23c3b260e6f2712]) successfully deleted
I20260929 16:41:24.759267 2418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:24.747723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:50233 (local address 127.0.197.66:41639)
0929 16:41:24.748002 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:24.748012 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:24.749723 (+ 1711us) server_negotiation.cc:407] Connection header received
0929 16:41:24.750065 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:24.750075 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:24.750180 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:24.750258 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:41:24.751346 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:24.752771 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:24.756578 (+ 3807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:24.758038 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:24.758207 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:24.758298 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:24.758743 (+ 445us) server_negotiation.cc:299] Negotiation successful
0929 16:41:24.758899 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 16:41:24.759548 2420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:24.747809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:38707 (local address 127.0.197.65:38669)
0929 16:41:24.747972 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:24.747979 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:24.752622 (+ 4643us) server_negotiation.cc:407] Connection header received
0929 16:41:24.752772 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:24.752778 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:24.752864 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:24.752997 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:41:24.754048 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:24.754964 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:24.757087 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:24.758333 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:24.758449 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:24.758496 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:24.759149 (+ 653us) server_negotiation.cc:299] Negotiation successful
0929 16:41:24.759299 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 16:41:24.759575 2453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:24.747363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.66:41639 (local address 127.0.197.67:50233)
0929 16:41:24.749584 (+ 2221us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:24.749612 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:24.749869 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:24.750534 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:24.750541 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:24.750576 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:41:24.751089 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:24.751107 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:24.752926 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:24.752932 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:24.756375 (+ 3443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:24.756390 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:24.758279 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:24.758286 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:24.758472 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:24.758510 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 16:41:24.758689 (+ 179us) client_negotiation.cc:240] Negotiation successful
0929 16:41:24.758860 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2038,"thread_start_us":4755,"threads_started":1}
I20260929 16:41:24.760527 2452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:24.747521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:38669 (local address 127.0.197.67:38707)
0929 16:41:24.752492 (+ 4971us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:24.752513 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:24.752645 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:24.753039 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:24.753045 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:24.753071 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:41:24.753840 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:24.753854 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:24.755174 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:24.755178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:24.756882 (+ 1704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:24.756908 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:24.758735 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:24.758741 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:24.758941 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:24.758975 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 16:41:24.760046 (+ 1071us) client_negotiation.cc:240] Negotiation successful
0929 16:41:24.760225 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4831,"thread_start_us":3731,"threads_started":1}
I20260929 16:41:24.760428 2214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d64154ac1ba448d876f47f50221d313" candidate_uuid: "5c2a05b56511413a8bb0d0ce168d689e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46bb3be715984c2aa30664a8fb35cf7b" is_pre_election: true
I20260929 16:41:24.760994 2214 raft_consensus.cc:2454] T 3d64154ac1ba448d876f47f50221d313 P 46bb3be715984c2aa30664a8fb35cf7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c2a05b56511413a8bb0d0ce168d689e in term 0.
I20260929 16:41:24.761248 2078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d64154ac1ba448d876f47f50221d313" candidate_uuid: "5c2a05b56511413a8bb0d0ce168d689e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0a80dcdaf6d4a299f3aefbd155e3783" is_pre_election: true
I20260929 16:41:24.761485 2078 raft_consensus.cc:2454] T 3d64154ac1ba448d876f47f50221d313 P f0a80dcdaf6d4a299f3aefbd155e3783 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c2a05b56511413a8bb0d0ce168d689e in term 0.
I20260929 16:41:24.761781 2286 leader_election.cc:302] T 3d64154ac1ba448d876f47f50221d313 P 5c2a05b56511413a8bb0d0ce168d689e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46bb3be715984c2aa30664a8fb35cf7b, 5c2a05b56511413a8bb0d0ce168d689e; no voters:
I20260929 16:41:24.762275 2431 raft_consensus.cc:2781] T 3d64154ac1ba448d876f47f50221d313 P 5c2a05b56511413a8bb0d0ce168d689e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:41:24.762411 2431 raft_consensus.cc:482] T 3d64154ac1ba448d876f47f50221d313 P 5c2a05b56511413a8bb0d0ce168d689e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:41:24.762516 2431 raft_consensus.cc:3037] T 3d64154ac1ba448d876f47f50221d313 P 5c2a05b56511413a8bb0d0ce168d689e [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:41:24.765373 2431 raft_consensus.cc:504] T 3d64154ac1ba448d876f47f50221d313 P 5c2a05b56511413a8bb0d0ce168d689e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a80dcdaf6d4a299f3aefbd155e3783" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 38669 } } peers { permanent_uuid: "46bb3be715984c2aa30664a8fb35cf7b" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 41639 } } peers { permanent_uuid: "5c2a05b56511413a8bb0d0ce168d689e" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33555 } }
I20260929 16:41:24.766074 2431 leader_election.cc:288] T 3d64154ac1ba448d876f47f50221d313 P 5c2a05b56511413a8bb0d0ce168d689e [CANDIDATE]: Term 1 election: Requested vote from peers f0a80dcdaf6d4a299f3aefbd155e3783 (127.0.197.65:38669), 46bb3be715984c2aa30664a8fb35cf7b (127.0.197.66:41639)
I20260929 16:41:24.766654 2214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d64154ac1ba448d876f47f50221d313" candidate_uuid: "5c2a05b56511413a8bb0d0ce168d689e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46bb3be715984c2aa30664a8fb35cf7b"
I20260929 16:41:24.766659 2078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d64154ac1ba448d876f47f50221d313" candidate_uuid: "5c2a05b56511413a8bb0d0ce168d689e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0a80dcdaf6d4a299f3aefbd155e3783"
I20260929 16:41:24.766883 2078 raft_consensus.cc:3037] T 3d64154ac1ba448d876f47f50221d313 P f0a80dcdaf6d4a299f3aefbd155e3783 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:41:24.766889 2214 raft_consensus.cc:3037] T 3d64154ac1ba448d876f47f50221d313 P 46bb3be715984c2aa30664a8fb35cf7b [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:41:24.770197 2078 raft_consensus.cc:2454] T 3d64154ac1ba448d876f47f50221d313 P f0a80dcdaf6d4a299f3aefbd155e3783 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c2a05b56511413a8bb0d0ce168d689e in term 1.
I20260929 16:41:24.770392 2214 raft_consensus.cc:2454] T 3d64154ac1ba448d876f47f50221d313 P 46bb3be715984c2aa30664a8fb35cf7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c2a05b56511413a8bb0d0ce168d689e in term 1.
I20260929 16:41:24.770820 2287 leader_election.cc:302] T 3d64154ac1ba448d876f47f50221d313 P 5c2a05b56511413a8bb0d0ce168d689e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c2a05b56511413a8bb0d0ce168d689e, f0a80dcdaf6d4a299f3aefbd155e3783; no voters:
I20260929 16:41:24.771344 2431 raft_consensus.cc:2781] T 3d64154ac1ba448d876f47f50221d313 P 5c2a05b56511413a8bb0d0ce168d689e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:41:24.771839 2431 raft_consensus.cc:686] T 3d64154ac1ba448d876f47f50221d313 P 5c2a05b56511413a8bb0d0ce168d689e [term 1 LEADER]: Becoming Leader. State: Replica: 5c2a05b56511413a8bb0d0ce168d689e, State: Running, Role: LEADER
I20260929 16:41:24.772284 2431 consensus_queue.cc:237] T 3d64154ac1ba448d876f47f50221d313 P 5c2a05b56511413a8bb0d0ce168d689e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a80dcdaf6d4a299f3aefbd155e3783" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 38669 } } peers { permanent_uuid: "46bb3be715984c2aa30664a8fb35cf7b" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 41639 } } peers { permanent_uuid: "5c2a05b56511413a8bb0d0ce168d689e" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33555 } }
I20260929 16:41:24.776258 1916 catalog_manager.cc:5495] T 3d64154ac1ba448d876f47f50221d313 P 5c2a05b56511413a8bb0d0ce168d689e reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c2a05b56511413a8bb0d0ce168d689e (127.0.197.67). New cstate: current_term: 1 leader_uuid: "5c2a05b56511413a8bb0d0ce168d689e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a80dcdaf6d4a299f3aefbd155e3783" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 38669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46bb3be715984c2aa30664a8fb35cf7b" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c2a05b56511413a8bb0d0ce168d689e" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33555 } health_report { overall_health: HEALTHY } } }
Sep 29 16:41:24 dist-test-slave-l31f krb5kdc[1872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:41:24.832751 2462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:24.814099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37717 (local address 127.0.0.1:38634)
0929 16:41:24.814895 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:24.814928 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:24.815580 (+ 652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:24.816127 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:24.816133 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:24.817246 (+ 1113us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:41:24.817572 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:24.817587 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:24.819145 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:24.819150 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:24.820800 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:24.820818 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:24.821647 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:24.821654 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:24.821847 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:24.822690 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:41:24.822715 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:41:24.825640 (+ 2925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:41:24.829775 (+ 4135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:24.829804 (+ 29us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:24.829812 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:24.830290 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:24.830720 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:24.830726 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:24.830729 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:24.830885 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:24.831518 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:41:24.831527 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:41:24.831698 (+ 171us) client_negotiation.cc:769] Sending connection context
0929 16:41:24.831892 (+ 194us) client_negotiation.cc:240] Negotiation successful
0929 16:41:24.832276 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"mutex_wait_us":5,"thread_start_us":456,"threads_started":1}
I20260929 16:41:24.832788 2461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:24.814164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38634 (local address 127.0.197.126:37717)
0929 16:41:24.815766 (+ 1602us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:24.815770 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:24.815786 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:41:24.815825 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:24.815828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:24.815878 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:24.816021 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:24.817804 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:24.818927 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:24.821012 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:24.821444 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:24.821555 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:24.825920 (+ 4365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:24.825926 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:24.825929 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:24.825962 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:24.829540 (+ 3578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:24.830439 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:24.830447 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:24.830451 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:24.830597 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:24.831001 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:24.831044 (+ 43us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:24.831048 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:24.831358 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:24.831537 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:24.831971 (+ 434us) server_negotiation.cc:299] Negotiation successful
0929 16:41:24.832413 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":209,"server-negotiator.queue_time_us":1464,"thread_start_us":826,"threads_started":1}
I20260929 16:41:24.839377 1916 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:38634:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 16:41:24.840498 1916 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:41:24.848948 2053 tablet_service.cc:1467] Processing CreateTablet for tablet 1eafb872de9646eebfcbda44b3cffadb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a514a33d94694921933edb97ce516acd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 16:41:24.849267 2333 tablet_service.cc:1467] Processing CreateTablet for tablet 1eafb872de9646eebfcbda44b3cffadb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a514a33d94694921933edb97ce516acd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 16:41:24.849635 2053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1eafb872de9646eebfcbda44b3cffadb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:41:24.849808 2333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1eafb872de9646eebfcbda44b3cffadb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:41:24.856567 2425 tablet_bootstrap.cc:492] T 1eafb872de9646eebfcbda44b3cffadb P f0a80dcdaf6d4a299f3aefbd155e3783: Bootstrap starting.
I20260929 16:41:24.858124 2425 tablet_bootstrap.cc:654] T 1eafb872de9646eebfcbda44b3cffadb P f0a80dcdaf6d4a299f3aefbd155e3783: Neither blocks nor log segments found. Creating new log.
I20260929 16:41:24.857899 2194 tablet_service.cc:1467] Processing CreateTablet for tablet 1eafb872de9646eebfcbda44b3cffadb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a514a33d94694921933edb97ce516acd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 16:41:24.858598 2194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1eafb872de9646eebfcbda44b3cffadb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:41:24.868337 2423 tablet_bootstrap.cc:492] T 1eafb872de9646eebfcbda44b3cffadb P 46bb3be715984c2aa30664a8fb35cf7b: Bootstrap starting.
I20260929 16:41:24.870048 2426 tablet_bootstrap.cc:492] T 1eafb872de9646eebfcbda44b3cffadb P 5c2a05b56511413a8bb0d0ce168d689e: Bootstrap starting.
I20260929 16:41:24.870227 2425 tablet_bootstrap.cc:492] T 1eafb872de9646eebfcbda44b3cffadb P f0a80dcdaf6d4a299f3aefbd155e3783: No bootstrap required, opened a new log
I20260929 16:41:24.870425 2425 ts_tablet_manager.cc:1397] T 1eafb872de9646eebfcbda44b3cffadb P f0a80dcdaf6d4a299f3aefbd155e3783: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.011s
I20260929 16:41:24.870456 2423 tablet_bootstrap.cc:654] T 1eafb872de9646eebfcbda44b3cffadb P 46bb3be715984c2aa30664a8fb35cf7b: Neither blocks nor log segments found. Creating new log.
I20260929 16:41:24.871281 2425 raft_consensus.cc:348] T 1eafb872de9646eebfcbda44b3cffadb P f0a80dcdaf6d4a299f3aefbd155e3783 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a80dcdaf6d4a299f3aefbd155e3783" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 38669 } } peers { permanent_uuid: "46bb3be715984c2aa30664a8fb35cf7b" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 41639 } } peers { permanent_uuid: "5c2a05b56511413a8bb0d0ce168d689e" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33555 } }
I20260929 16:41:24.871554 2425 raft_consensus.cc:374] T 1eafb872de9646eebfcbda44b3cffadb P f0a80dcdaf6d4a299f3aefbd155e3783 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:41:24.871636 2425 raft_consensus.cc:729] T 1eafb872de9646eebfcbda44b3cffadb P f0a80dcdaf6d4a299f3aefbd155e3783 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0a80dcdaf6d4a299f3aefbd155e3783, State: Initialized, Role: FOLLOWER
I20260929 16:41:24.871896 2425 consensus_queue.cc:260] T 1eafb872de9646eebfcbda44b3cffadb P f0a80dcdaf6d4a299f3aefbd155e3783 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a80dcdaf6d4a299f3aefbd155e3783" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 38669 } } peers { permanent_uuid: "46bb3be715984c2aa30664a8fb35cf7b" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 41639 } } peers { permanent_uuid: "5c2a05b56511413a8bb0d0ce168d689e" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33555 } }
I20260929 16:41:24.872066 2426 tablet_bootstrap.cc:654] T 1eafb872de9646eebfcbda44b3cffadb P 5c2a05b56511413a8bb0d0ce168d689e: Neither blocks nor log segments found. Creating new log.
I20260929 16:41:24.872712 2425 ts_tablet_manager.cc:1428] T 1eafb872de9646eebfcbda44b3cffadb P f0a80dcdaf6d4a299f3aefbd155e3783: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 16:41:24.872449 2430 tablet_replica.cc:439] T 1eafb872de9646eebfcbda44b3cffadb P f0a80dcdaf6d4a299f3aefbd155e3783: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a80dcdaf6d4a299f3aefbd155e3783" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 38669 } } peers { permanent_uuid: "46bb3be715984c2aa30664a8fb35cf7b" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 41639 } } peers { permanent_uuid: "5c2a05b56511413a8bb0d0ce168d689e" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33555 } } }
I20260929 16:41:24.873008 2430 tablet_replica.cc:442] T 1eafb872de9646eebfcbda44b3cffadb P f0a80dcdaf6d4a299f3aefbd155e3783: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:41:24.874975 2426 tablet_bootstrap.cc:492] T 1eafb872de9646eebfcbda44b3cffadb P 5c2a05b56511413a8bb0d0ce168d689e: No bootstrap required, opened a new log
I20260929 16:41:24.875200 2426 ts_tablet_manager.cc:1397] T 1eafb872de9646eebfcbda44b3cffadb P 5c2a05b56511413a8bb0d0ce168d689e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260929 16:41:24.876008 2426 raft_consensus.cc:348] T 1eafb872de9646eebfcbda44b3cffadb P 5c2a05b56511413a8bb0d0ce168d689e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a80dcdaf6d4a299f3aefbd155e3783" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 38669 } } peers { permanent_uuid: "46bb3be715984c2aa30664a8fb35cf7b" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 41639 } } peers { permanent_uuid: "5c2a05b56511413a8bb0d0ce168d689e" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33555 } }
I20260929 16:41:24.876267 2426 raft_consensus.cc:374] T 1eafb872de9646eebfcbda44b3cffadb P 5c2a05b56511413a8bb0d0ce168d689e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:41:24.876345 2426 raft_consensus.cc:729] T 1eafb872de9646eebfcbda44b3cffadb P 5c2a05b56511413a8bb0d0ce168d689e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c2a05b56511413a8bb0d0ce168d689e, State: Initialized, Role: FOLLOWER
I20260929 16:41:24.876587 2426 consensus_queue.cc:260] T 1eafb872de9646eebfcbda44b3cffadb P 5c2a05b56511413a8bb0d0ce168d689e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a80dcdaf6d4a299f3aefbd155e3783" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 38669 } } peers { permanent_uuid: "46bb3be715984c2aa30664a8fb35cf7b" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 41639 } } peers { permanent_uuid: "5c2a05b56511413a8bb0d0ce168d689e" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33555 } }
I20260929 16:41:24.877076 2431 tablet_replica.cc:439] T 1eafb872de9646eebfcbda44b3cffadb P 5c2a05b56511413a8bb0d0ce168d689e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a80dcdaf6d4a299f3aefbd155e3783" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 38669 } } peers { permanent_uuid: "46bb3be715984c2aa30664a8fb35cf7b" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 41639 } } peers { permanent_uuid: "5c2a05b56511413a8bb0d0ce168d689e" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33555 } } }
I20260929 16:41:24.877249 2426 ts_tablet_manager.cc:1428] T 1eafb872de9646eebfcbda44b3cffadb P 5c2a05b56511413a8bb0d0ce168d689e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 16:41:24.877375 2431 tablet_replica.cc:442] T 1eafb872de9646eebfcbda44b3cffadb P 5c2a05b56511413a8bb0d0ce168d689e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:41:24.879190 2423 tablet_bootstrap.cc:492] T 1eafb872de9646eebfcbda44b3cffadb P 46bb3be715984c2aa30664a8fb35cf7b: No bootstrap required, opened a new log
I20260929 16:41:24.879451 2423 ts_tablet_manager.cc:1397] T 1eafb872de9646eebfcbda44b3cffadb P 46bb3be715984c2aa30664a8fb35cf7b: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.001s
I20260929 16:41:24.880375 2423 raft_consensus.cc:348] T 1eafb872de9646eebfcbda44b3cffadb P 46bb3be715984c2aa30664a8fb35cf7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a80dcdaf6d4a299f3aefbd155e3783" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 38669 } } peers { permanent_uuid: "46bb3be715984c2aa30664a8fb35cf7b" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 41639 } } peers { permanent_uuid: "5c2a05b56511413a8bb0d0ce168d689e" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33555 } }
I20260929 16:41:24.880628 2423 raft_consensus.cc:374] T 1eafb872de9646eebfcbda44b3cffadb P 46bb3be715984c2aa30664a8fb35cf7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:41:24.880738 2423 raft_consensus.cc:729] T 1eafb872de9646eebfcbda44b3cffadb P 46bb3be715984c2aa30664a8fb35cf7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46bb3be715984c2aa30664a8fb35cf7b, State: Initialized, Role: FOLLOWER
I20260929 16:41:24.881069 2423 consensus_queue.cc:260] T 1eafb872de9646eebfcbda44b3cffadb P 46bb3be715984c2aa30664a8fb35cf7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a80dcdaf6d4a299f3aefbd155e3783" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 38669 } } peers { permanent_uuid: "46bb3be715984c2aa30664a8fb35cf7b" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 41639 } } peers { permanent_uuid: "5c2a05b56511413a8bb0d0ce168d689e" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33555 } }
I20260929 16:41:24.881743 2423 ts_tablet_manager.cc:1428] T 1eafb872de9646eebfcbda44b3cffadb P 46bb3be715984c2aa30664a8fb35cf7b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 16:41:24.881611 2429 tablet_replica.cc:439] T 1eafb872de9646eebfcbda44b3cffadb P 46bb3be715984c2aa30664a8fb35cf7b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a80dcdaf6d4a299f3aefbd155e3783" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 38669 } } peers { permanent_uuid: "46bb3be715984c2aa30664a8fb35cf7b" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 41639 } } peers { permanent_uuid: "5c2a05b56511413a8bb0d0ce168d689e" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33555 } } }
I20260929 16:41:24.881911 2429 tablet_replica.cc:442] T 1eafb872de9646eebfcbda44b3cffadb P 46bb3be715984c2aa30664a8fb35cf7b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:41:24.906395 2429 raft_consensus.cc:482] T 1eafb872de9646eebfcbda44b3cffadb P 46bb3be715984c2aa30664a8fb35cf7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:41:24.906634 2429 raft_consensus.cc:504] T 1eafb872de9646eebfcbda44b3cffadb P 46bb3be715984c2aa30664a8fb35cf7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a80dcdaf6d4a299f3aefbd155e3783" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 38669 } } peers { permanent_uuid: "46bb3be715984c2aa30664a8fb35cf7b" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 41639 } } peers { permanent_uuid: "5c2a05b56511413a8bb0d0ce168d689e" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33555 } }
I20260929 16:41:24.907475 2429 leader_election.cc:288] T 1eafb872de9646eebfcbda44b3cffadb P 46bb3be715984c2aa30664a8fb35cf7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0a80dcdaf6d4a299f3aefbd155e3783 (127.0.197.65:38669), 5c2a05b56511413a8bb0d0ce168d689e (127.0.197.67:33555)
I20260929 16:41:24.908085 2078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eafb872de9646eebfcbda44b3cffadb" candidate_uuid: "46bb3be715984c2aa30664a8fb35cf7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0a80dcdaf6d4a299f3aefbd155e3783" is_pre_election: true
I20260929 16:41:24.908129 2354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eafb872de9646eebfcbda44b3cffadb" candidate_uuid: "46bb3be715984c2aa30664a8fb35cf7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c2a05b56511413a8bb0d0ce168d689e" is_pre_election: true
I20260929 16:41:24.908358 2078 raft_consensus.cc:2454] T 1eafb872de9646eebfcbda44b3cffadb P f0a80dcdaf6d4a299f3aefbd155e3783 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46bb3be715984c2aa30664a8fb35cf7b in term 0.
I20260929 16:41:24.908383 2354 raft_consensus.cc:2454] T 1eafb872de9646eebfcbda44b3cffadb P 5c2a05b56511413a8bb0d0ce168d689e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46bb3be715984c2aa30664a8fb35cf7b in term 0.
I20260929 16:41:24.908821 2146 leader_election.cc:302] T 1eafb872de9646eebfcbda44b3cffadb P 46bb3be715984c2aa30664a8fb35cf7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46bb3be715984c2aa30664a8fb35cf7b, 5c2a05b56511413a8bb0d0ce168d689e; no voters:
I20260929 16:41:24.909148 2429 raft_consensus.cc:2781] T 1eafb872de9646eebfcbda44b3cffadb P 46bb3be715984c2aa30664a8fb35cf7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:41:24.909288 2429 raft_consensus.cc:482] T 1eafb872de9646eebfcbda44b3cffadb P 46bb3be715984c2aa30664a8fb35cf7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:41:24.909389 2429 raft_consensus.cc:3037] T 1eafb872de9646eebfcbda44b3cffadb P 46bb3be715984c2aa30664a8fb35cf7b [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:41:24.913122 2429 raft_consensus.cc:504] T 1eafb872de9646eebfcbda44b3cffadb P 46bb3be715984c2aa30664a8fb35cf7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a80dcdaf6d4a299f3aefbd155e3783" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 38669 } } peers { permanent_uuid: "46bb3be715984c2aa30664a8fb35cf7b" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 41639 } } peers { permanent_uuid: "5c2a05b56511413a8bb0d0ce168d689e" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33555 } }
I20260929 16:41:24.913908 2429 leader_election.cc:288] T 1eafb872de9646eebfcbda44b3cffadb P 46bb3be715984c2aa30664a8fb35cf7b [CANDIDATE]: Term 1 election: Requested vote from peers f0a80dcdaf6d4a299f3aefbd155e3783 (127.0.197.65:38669), 5c2a05b56511413a8bb0d0ce168d689e (127.0.197.67:33555)
I20260929 16:41:24.914600 2354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eafb872de9646eebfcbda44b3cffadb" candidate_uuid: "46bb3be715984c2aa30664a8fb35cf7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c2a05b56511413a8bb0d0ce168d689e"
I20260929 16:41:24.914583 2078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eafb872de9646eebfcbda44b3cffadb" candidate_uuid: "46bb3be715984c2aa30664a8fb35cf7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0a80dcdaf6d4a299f3aefbd155e3783"
I20260929 16:41:24.914850 2354 raft_consensus.cc:3037] T 1eafb872de9646eebfcbda44b3cffadb P 5c2a05b56511413a8bb0d0ce168d689e [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:41:24.914867 2078 raft_consensus.cc:3037] T 1eafb872de9646eebfcbda44b3cffadb P f0a80dcdaf6d4a299f3aefbd155e3783 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:41:24.917057 2354 raft_consensus.cc:2454] T 1eafb872de9646eebfcbda44b3cffadb P 5c2a05b56511413a8bb0d0ce168d689e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46bb3be715984c2aa30664a8fb35cf7b in term 1.
I20260929 16:41:24.917740 2146 leader_election.cc:302] T 1eafb872de9646eebfcbda44b3cffadb P 46bb3be715984c2aa30664a8fb35cf7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46bb3be715984c2aa30664a8fb35cf7b, 5c2a05b56511413a8bb0d0ce168d689e; no voters:
I20260929 16:41:24.917989 2078 raft_consensus.cc:2454] T 1eafb872de9646eebfcbda44b3cffadb P f0a80dcdaf6d4a299f3aefbd155e3783 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46bb3be715984c2aa30664a8fb35cf7b in term 1.
I20260929 16:41:24.918435 2429 raft_consensus.cc:2781] T 1eafb872de9646eebfcbda44b3cffadb P 46bb3be715984c2aa30664a8fb35cf7b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:41:24.918608 2429 raft_consensus.cc:686] T 1eafb872de9646eebfcbda44b3cffadb P 46bb3be715984c2aa30664a8fb35cf7b [term 1 LEADER]: Becoming Leader. State: Replica: 46bb3be715984c2aa30664a8fb35cf7b, State: Running, Role: LEADER
I20260929 16:41:24.918897 2429 consensus_queue.cc:237] T 1eafb872de9646eebfcbda44b3cffadb P 46bb3be715984c2aa30664a8fb35cf7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a80dcdaf6d4a299f3aefbd155e3783" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 38669 } } peers { permanent_uuid: "46bb3be715984c2aa30664a8fb35cf7b" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 41639 } } peers { permanent_uuid: "5c2a05b56511413a8bb0d0ce168d689e" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33555 } }
I20260929 16:41:24.920966 2434 tablet_replica.cc:439] T 1eafb872de9646eebfcbda44b3cffadb P 46bb3be715984c2aa30664a8fb35cf7b: TxnStatusTablet state changed. Reason: New leader 46bb3be715984c2aa30664a8fb35cf7b. Latest consensus state: current_term: 1 leader_uuid: "46bb3be715984c2aa30664a8fb35cf7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a80dcdaf6d4a299f3aefbd155e3783" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 38669 } } peers { permanent_uuid: "46bb3be715984c2aa30664a8fb35cf7b" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 41639 } } peers { permanent_uuid: "5c2a05b56511413a8bb0d0ce168d689e" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33555 } } }
I20260929 16:41:24.921331 2434 tablet_replica.cc:442] T 1eafb872de9646eebfcbda44b3cffadb P 46bb3be715984c2aa30664a8fb35cf7b: This TxnStatusTablet replica's current role is: LEADER
I20260929 16:41:24.922010 2467 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 16:41:24.922685 2467 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 16:41:24.922777 1916 catalog_manager.cc:5495] T 1eafb872de9646eebfcbda44b3cffadb P 46bb3be715984c2aa30664a8fb35cf7b reported cstate change: term changed from 0 to 1, leader changed from <none> to 46bb3be715984c2aa30664a8fb35cf7b (127.0.197.66). New cstate: current_term: 1 leader_uuid: "46bb3be715984c2aa30664a8fb35cf7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a80dcdaf6d4a299f3aefbd155e3783" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 38669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46bb3be715984c2aa30664a8fb35cf7b" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c2a05b56511413a8bb0d0ce168d689e" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33555 } health_report { overall_health: UNKNOWN } } }
I20260929 16:41:24.957593 2462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:24.951684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.66:41639 (local address 127.0.0.1:45378)
0929 16:41:24.951915 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:24.951943 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:24.952102 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:24.952560 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:24.952565 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:24.952580 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:41:24.952807 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:24.952908 (+ 101us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:24.954036 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:24.954043 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:24.955286 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:24.955301 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:24.956174 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:24.956179 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:24.956353 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:24.956420 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:41:24.956915 (+ 495us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:41:24.956956 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 16:41:24.957055 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 16:41:24.957219 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 16:41:24.957674 2418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:24.951965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45378 (local address 127.0.197.66:41639)
0929 16:41:24.952199 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:24.952209 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:24.952230 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:41:24.952336 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:24.952340 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:24.952406 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:24.952511 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:41:24.953099 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:24.953877 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:24.955455 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:24.956001 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:24.956153 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:24.956628 (+ 475us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:41:24.956757 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:41:24.956888 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:24.957082 (+ 194us) server_negotiation.cc:299] Negotiation successful
0929 16:41:24.957167 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 16:41:24.962373 2078 raft_consensus.cc:1261] T 1eafb872de9646eebfcbda44b3cffadb P f0a80dcdaf6d4a299f3aefbd155e3783 [term 1 FOLLOWER]: Refusing update from remote peer 46bb3be715984c2aa30664a8fb35cf7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:41:24.962718 2354 raft_consensus.cc:1261] T 1eafb872de9646eebfcbda44b3cffadb P 5c2a05b56511413a8bb0d0ce168d689e [term 1 FOLLOWER]: Refusing update from remote peer 46bb3be715984c2aa30664a8fb35cf7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:41:24.963196 2429 consensus_queue.cc:1035] T 1eafb872de9646eebfcbda44b3cffadb P 46bb3be715984c2aa30664a8fb35cf7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a80dcdaf6d4a299f3aefbd155e3783" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 38669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:41:24.963478 2434 consensus_queue.cc:1035] T 1eafb872de9646eebfcbda44b3cffadb P 46bb3be715984c2aa30664a8fb35cf7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c2a05b56511413a8bb0d0ce168d689e" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:41:24.967532 2431 tablet_replica.cc:439] T 1eafb872de9646eebfcbda44b3cffadb P 5c2a05b56511413a8bb0d0ce168d689e: TxnStatusTablet state changed. Reason: New leader 46bb3be715984c2aa30664a8fb35cf7b. Latest consensus state: current_term: 1 leader_uuid: "46bb3be715984c2aa30664a8fb35cf7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a80dcdaf6d4a299f3aefbd155e3783" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 38669 } } peers { permanent_uuid: "46bb3be715984c2aa30664a8fb35cf7b" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 41639 } } peers { permanent_uuid: "5c2a05b56511413a8bb0d0ce168d689e" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33555 } } }
I20260929 16:41:24.967873 2431 tablet_replica.cc:442] T 1eafb872de9646eebfcbda44b3cffadb P 5c2a05b56511413a8bb0d0ce168d689e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:41:24.971913 2429 tablet_replica.cc:439] T 1eafb872de9646eebfcbda44b3cffadb P 46bb3be715984c2aa30664a8fb35cf7b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "46bb3be715984c2aa30664a8fb35cf7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a80dcdaf6d4a299f3aefbd155e3783" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 38669 } } peers { permanent_uuid: "46bb3be715984c2aa30664a8fb35cf7b" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 41639 } } peers { permanent_uuid: "5c2a05b56511413a8bb0d0ce168d689e" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33555 } } }
I20260929 16:41:24.972628 2429 tablet_replica.cc:442] T 1eafb872de9646eebfcbda44b3cffadb P 46bb3be715984c2aa30664a8fb35cf7b: This TxnStatusTablet replica's current role is: LEADER
I20260929 16:41:24.972937 2437 mvcc.cc:205] Tried to move back new op lower bound from 7334707547998011392 to 7334707547830870016. Current Snapshot: MvccSnapshot[applied={T|T < 7334707547998011392 or (T in {7334707547998011392})}]
I20260929 16:41:24.978796 2431 tablet_replica.cc:439] T 1eafb872de9646eebfcbda44b3cffadb P 5c2a05b56511413a8bb0d0ce168d689e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "46bb3be715984c2aa30664a8fb35cf7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a80dcdaf6d4a299f3aefbd155e3783" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 38669 } } peers { permanent_uuid: "46bb3be715984c2aa30664a8fb35cf7b" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 41639 } } peers { permanent_uuid: "5c2a05b56511413a8bb0d0ce168d689e" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33555 } } }
I20260929 16:41:24.979166 2431 tablet_replica.cc:442] T 1eafb872de9646eebfcbda44b3cffadb P 5c2a05b56511413a8bb0d0ce168d689e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:41:24.981390 2430 tablet_replica.cc:439] T 1eafb872de9646eebfcbda44b3cffadb P f0a80dcdaf6d4a299f3aefbd155e3783: TxnStatusTablet state changed. Reason: New leader 46bb3be715984c2aa30664a8fb35cf7b. Latest consensus state: current_term: 1 leader_uuid: "46bb3be715984c2aa30664a8fb35cf7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a80dcdaf6d4a299f3aefbd155e3783" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 38669 } } peers { permanent_uuid: "46bb3be715984c2aa30664a8fb35cf7b" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 41639 } } peers { permanent_uuid: "5c2a05b56511413a8bb0d0ce168d689e" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33555 } } }
I20260929 16:41:24.981709 2430 tablet_replica.cc:442] T 1eafb872de9646eebfcbda44b3cffadb P f0a80dcdaf6d4a299f3aefbd155e3783: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:41:24.984337 2434 tablet_replica.cc:439] T 1eafb872de9646eebfcbda44b3cffadb P 46bb3be715984c2aa30664a8fb35cf7b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "46bb3be715984c2aa30664a8fb35cf7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a80dcdaf6d4a299f3aefbd155e3783" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 38669 } } peers { permanent_uuid: "46bb3be715984c2aa30664a8fb35cf7b" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 41639 } } peers { permanent_uuid: "5c2a05b56511413a8bb0d0ce168d689e" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33555 } } }
I20260929 16:41:24.984637 2434 tablet_replica.cc:442] T 1eafb872de9646eebfcbda44b3cffadb P 46bb3be715984c2aa30664a8fb35cf7b: This TxnStatusTablet replica's current role is: LEADER
I20260929 16:41:24.986582 2430 tablet_replica.cc:439] T 1eafb872de9646eebfcbda44b3cffadb P f0a80dcdaf6d4a299f3aefbd155e3783: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "46bb3be715984c2aa30664a8fb35cf7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a80dcdaf6d4a299f3aefbd155e3783" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 38669 } } peers { permanent_uuid: "46bb3be715984c2aa30664a8fb35cf7b" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 41639 } } peers { permanent_uuid: "5c2a05b56511413a8bb0d0ce168d689e" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33555 } } }
I20260929 16:41:24.986863 2430 tablet_replica.cc:442] T 1eafb872de9646eebfcbda44b3cffadb P f0a80dcdaf6d4a299f3aefbd155e3783: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:41:24.990219 2422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:24.981748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50476 (local address 127.0.197.67:33555)
0929 16:41:24.981906 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:24.981912 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:24.981930 (+ 18us) server_negotiation.cc:407] Connection header received
0929 16:41:24.982104 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:24.982108 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:24.982206 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:24.982355 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:41:24.983187 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:24.984354 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:24.986269 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:24.986950 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:24.987156 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:24.987892 (+ 736us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:41:24.988738 (+ 846us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:41:24.988909 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:24.989442 (+ 533us) server_negotiation.cc:299] Negotiation successful
0929 16:41:24.989537 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 16:41:25.030112 2418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:25.019487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:46311 (local address 127.0.197.66:41639)
0929 16:41:25.019693 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:25.019701 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:25.023761 (+ 4060us) server_negotiation.cc:407] Connection header received
0929 16:41:25.024059 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:25.024064 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:25.024139 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:25.024236 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:41:25.024889 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:25.025706 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:25.028060 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:25.028546 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:25.028673 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:25.029338 (+ 665us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:41:25.029494 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:41:25.029576 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:25.029770 (+ 194us) server_negotiation.cc:299] Negotiation successful
0929 16:41:25.029853 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 16:41:25.030318 2473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:25.019368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.66:41639 (local address 127.0.197.67:46311)
0929 16:41:25.023591 (+ 4223us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:25.023617 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:25.023844 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:25.024266 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:25.024271 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:25.024284 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:41:25.024680 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:25.024694 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:25.025860 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:25.025866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:25.027776 (+ 1910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:25.027796 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:25.028788 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:25.028796 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:25.028989 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:25.029089 (+ 100us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:41:25.029623 (+ 534us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:41:25.029636 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 16:41:25.029735 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 16:41:25.029908 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3993,"thread_start_us":3827,"threads_started":1}
I20260929 16:41:25.043205 2078 raft_consensus.cc:1261] T 3d64154ac1ba448d876f47f50221d313 P f0a80dcdaf6d4a299f3aefbd155e3783 [term 1 FOLLOWER]: Refusing update from remote peer 5c2a05b56511413a8bb0d0ce168d689e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:41:25.043320 2214 raft_consensus.cc:1261] T 3d64154ac1ba448d876f47f50221d313 P 46bb3be715984c2aa30664a8fb35cf7b [term 1 FOLLOWER]: Refusing update from remote peer 5c2a05b56511413a8bb0d0ce168d689e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:41:25.044188 2454 consensus_queue.cc:1035] T 3d64154ac1ba448d876f47f50221d313 P 5c2a05b56511413a8bb0d0ce168d689e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a80dcdaf6d4a299f3aefbd155e3783" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 38669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:41:25.045017 2454 consensus_queue.cc:1035] T 3d64154ac1ba448d876f47f50221d313 P 5c2a05b56511413a8bb0d0ce168d689e [LEADER]: Connected to new peer: Peer: permanent_uuid: "46bb3be715984c2aa30664a8fb35cf7b" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:41:25.052188 2438 mvcc.cc:205] Tried to move back new op lower bound from 7334707548326973440 to 7334707547232018432. Current Snapshot: MvccSnapshot[applied={T|T < 7334707548326973440}]
I20260929 16:41:25.060866 2439 mvcc.cc:205] Tried to move back new op lower bound from 7334707548326973440 to 7334707547232018432. Current Snapshot: MvccSnapshot[applied={T|T < 7334707548326973440}]
I20260929 16:41:25.096048 2422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:25.083673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:40899 (local address 127.0.197.67:33555)
0929 16:41:25.083944 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:25.083953 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:25.088017 (+ 4064us) server_negotiation.cc:407] Connection header received
0929 16:41:25.088447 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:25.088453 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:25.088545 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:25.088616 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:41:25.089350 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:25.090416 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:25.093592 (+ 3176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:25.094265 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:25.094388 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:25.095179 (+ 791us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:41:25.095337 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:41:25.095411 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:25.095664 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 16:41:25.095759 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 16:41:25.096975 2479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:25.083507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.67:33555 (local address 127.0.197.66:40899)
0929 16:41:25.087857 (+ 4350us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:25.087885 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:25.088235 (+ 350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:25.088873 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:25.088884 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:25.088899 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:41:25.089170 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:25.089183 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:25.090646 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:25.090652 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:25.093277 (+ 2625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:25.093336 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:25.094628 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:25.094636 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:25.094839 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:25.094914 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:41:25.095497 (+ 583us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:41:25.095522 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 16:41:25.096327 (+ 805us) client_negotiation.cc:240] Negotiation successful
0929 16:41:25.096499 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4025,"thread_start_us":3854,"threads_started":1}
I20260929 16:41:25.143711 1917 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:41:25.144028 1917 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:41:25.148093 1917 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b1e9d1fcf7c2470883bd4ca247fff193: Sending DeleteTablet for 3 replicas of tablet 3d64154ac1ba448d876f47f50221d313
I20260929 16:41:25.149211 2053 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d64154ac1ba448d876f47f50221d313 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:41:25 UTC) from {username='kudu', principal='kudu/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:58982
I20260929 16:41:25.149300 2194 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d64154ac1ba448d876f47f50221d313 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:41:25 UTC) from {username='kudu', principal='kudu/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:45364
I20260929 16:41:25.149555 2448 tablet_replica.cc:330] T 3d64154ac1ba448d876f47f50221d313 P f0a80dcdaf6d4a299f3aefbd155e3783: stopping tablet replica
I20260929 16:41:25.149592 2447 tablet_replica.cc:330] T 3d64154ac1ba448d876f47f50221d313 P 46bb3be715984c2aa30664a8fb35cf7b: stopping tablet replica
I20260929 16:41:25.149753 2448 raft_consensus.cc:2229] T 3d64154ac1ba448d876f47f50221d313 P f0a80dcdaf6d4a299f3aefbd155e3783 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:41:25.149791 2447 raft_consensus.cc:2229] T 3d64154ac1ba448d876f47f50221d313 P 46bb3be715984c2aa30664a8fb35cf7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:41:25.149963 2448 raft_consensus.cc:2258] T 3d64154ac1ba448d876f47f50221d313 P f0a80dcdaf6d4a299f3aefbd155e3783 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:41:25.149994 2447 raft_consensus.cc:2258] T 3d64154ac1ba448d876f47f50221d313 P 46bb3be715984c2aa30664a8fb35cf7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:41:25.150734 2333 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d64154ac1ba448d876f47f50221d313 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:41:25 UTC) from {username='kudu', principal='kudu/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:50466
I20260929 16:41:25.151074 2446 tablet_replica.cc:330] T 3d64154ac1ba448d876f47f50221d313 P 5c2a05b56511413a8bb0d0ce168d689e: stopping tablet replica
I20260929 16:41:25.151279 2446 raft_consensus.cc:2229] T 3d64154ac1ba448d876f47f50221d313 P 5c2a05b56511413a8bb0d0ce168d689e [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:41:25.151630 2448 ts_tablet_manager.cc:1905] T 3d64154ac1ba448d876f47f50221d313 P f0a80dcdaf6d4a299f3aefbd155e3783: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:41:25.151829 2446 raft_consensus.cc:2258] T 3d64154ac1ba448d876f47f50221d313 P 5c2a05b56511413a8bb0d0ce168d689e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:41:25.152860 2446 ts_tablet_manager.cc:1905] T 3d64154ac1ba448d876f47f50221d313 P 5c2a05b56511413a8bb0d0ce168d689e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:41:25.158201 2447 ts_tablet_manager.cc:1905] T 3d64154ac1ba448d876f47f50221d313 P 46bb3be715984c2aa30664a8fb35cf7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:41:25.160696 2448 ts_tablet_manager.cc:1918] T 3d64154ac1ba448d876f47f50221d313 P f0a80dcdaf6d4a299f3aefbd155e3783: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:41:25.160897 2448 log.cc:1196] T 3d64154ac1ba448d876f47f50221d313 P f0a80dcdaf6d4a299f3aefbd155e3783: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/ts-0/wal/wals/3d64154ac1ba448d876f47f50221d313
I20260929 16:41:25.161451 2448 ts_tablet_manager.cc:1939] T 3d64154ac1ba448d876f47f50221d313 P f0a80dcdaf6d4a299f3aefbd155e3783: Deleting consensus metadata
I20260929 16:41:25.161645 2446 ts_tablet_manager.cc:1918] T 3d64154ac1ba448d876f47f50221d313 P 5c2a05b56511413a8bb0d0ce168d689e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:41:25.161849 2446 log.cc:1196] T 3d64154ac1ba448d876f47f50221d313 P 5c2a05b56511413a8bb0d0ce168d689e: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/ts-2/wal/wals/3d64154ac1ba448d876f47f50221d313
I20260929 16:41:25.162338 2446 ts_tablet_manager.cc:1939] T 3d64154ac1ba448d876f47f50221d313 P 5c2a05b56511413a8bb0d0ce168d689e: Deleting consensus metadata
I20260929 16:41:25.162807 1904 catalog_manager.cc:4841] TS f0a80dcdaf6d4a299f3aefbd155e3783 (127.0.197.65:38669): tablet 3d64154ac1ba448d876f47f50221d313 (table test-table [id=59738d2a417a45b8bb435121ce8cada9]) successfully deleted
I20260929 16:41:25.163638 1903 catalog_manager.cc:4841] TS 5c2a05b56511413a8bb0d0ce168d689e (127.0.197.67:33555): tablet 3d64154ac1ba448d876f47f50221d313 (table test-table [id=59738d2a417a45b8bb435121ce8cada9]) successfully deleted
I20260929 16:41:25.169859 2447 ts_tablet_manager.cc:1918] T 3d64154ac1ba448d876f47f50221d313 P 46bb3be715984c2aa30664a8fb35cf7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:41:25.170835 2447 log.cc:1196] T 3d64154ac1ba448d876f47f50221d313 P 46bb3be715984c2aa30664a8fb35cf7b: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790700068255611-789-0/minicluster-data/ts-1/wal/wals/3d64154ac1ba448d876f47f50221d313
I20260929 16:41:25.171787 2447 ts_tablet_manager.cc:1939] T 3d64154ac1ba448d876f47f50221d313 P 46bb3be715984c2aa30664a8fb35cf7b: Deleting consensus metadata
I20260929 16:41:25.173364 1903 catalog_manager.cc:4841] TS 46bb3be715984c2aa30664a8fb35cf7b (127.0.197.66:41639): tablet 3d64154ac1ba448d876f47f50221d313 (table test-table [id=59738d2a417a45b8bb435121ce8cada9]) successfully deleted
Sep 29 16:41:25 dist-test-slave-l31f krb5kdc[1872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700084, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.197.65@KRBTEST.COM
I20260929 16:41:25.198732 2420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:25.163658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58984 (local address 127.0.197.65:38669)
0929 16:41:25.163859 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:25.163866 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:25.174182 (+ 10316us) server_negotiation.cc:407] Connection header received
0929 16:41:25.174492 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:25.174497 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:25.174587 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:25.174663 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:25.177115 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:25.178364 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:25.180848 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:25.181660 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:25.181793 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:25.188672 (+ 6879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:25.188695 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:25.188717 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:25.188780 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:25.193786 (+ 5006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:25.195265 (+ 1479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:25.195282 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:25.195286 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:25.196159 (+ 873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:25.196776 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:25.196783 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:25.196787 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:25.197306 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:25.197684 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:25.198042 (+ 358us) server_negotiation.cc:299] Negotiation successful
0929 16:41:25.198326 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260929 16:41:25.199731 2119 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58984
I20260929 16:41:25.229346 2461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:25.206042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38642 (local address 127.0.197.126:37717)
0929 16:41:25.206253 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:25.206261 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:25.207685 (+ 1424us) server_negotiation.cc:407] Connection header received
0929 16:41:25.207897 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:25.207903 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:25.207997 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:25.208127 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:25.210203 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:25.211428 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:25.213487 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:25.214227 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:25.214631 (+ 404us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:25.219464 (+ 4833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:25.219486 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:25.219491 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:25.219553 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:25.225393 (+ 5840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:25.226463 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:25.226469 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:25.226484 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:25.226656 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:25.227157 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:25.227163 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:25.227167 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:25.227825 (+ 658us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:25.228089 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:25.228505 (+ 416us) server_negotiation.cc:299] Negotiation successful
0929 16:41:25.228985 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260929 16:41:25.230188 1917 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38642
I20260929 16:41:25.236771 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 1988
I20260929 16:41:25.266376 2420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:25.239380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58998 (local address 127.0.197.65:38669)
0929 16:41:25.239592 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:25.239599 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:25.241442 (+ 1843us) server_negotiation.cc:407] Connection header received
0929 16:41:25.241557 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:25.241562 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:25.241657 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:25.241785 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:25.243548 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:25.244758 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:25.246783 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:25.247531 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:25.247653 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:25.258007 (+ 10354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:25.258017 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:25.258023 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:25.258078 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:25.262468 (+ 4390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:25.263801 (+ 1333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:25.263808 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:25.263812 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:25.264031 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:25.264528 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:25.264533 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:25.264537 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:25.264899 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:25.265158 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:25.265716 (+ 558us) server_negotiation.cc:299] Negotiation successful
0929 16:41:25.265985 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260929 16:41:25.268119 2119 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58998
W20260929 16:41:25.268836 789 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 1988: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:41:25.269040 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 1988
I20260929 16:41:25.291623 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 2128
Sep 29 16:41:25 dist-test-slave-l31f krb5kdc[1872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700084, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.197.66@KRBTEST.COM
I20260929 16:41:25.313205 2418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:25.292859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45392 (local address 127.0.197.66:41639)
0929 16:41:25.293040 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:25.293047 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:25.293292 (+ 245us) server_negotiation.cc:407] Connection header received
0929 16:41:25.293623 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:25.293629 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:25.293709 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:25.293777 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:25.295950 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:25.297092 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:25.298998 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:25.299664 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:25.299764 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:25.305604 (+ 5840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:25.305626 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:25.305650 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:25.305704 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:25.309555 (+ 3851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:25.310556 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:25.310566 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:25.310570 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:25.310738 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:25.311200 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:25.311205 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:25.311209 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:25.311674 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:25.311969 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:25.312564 (+ 595us) server_negotiation.cc:299] Negotiation successful
0929 16:41:25.312858 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260929 16:41:25.315129 2259 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45392
W20260929 16:41:25.316016 789 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 2128: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:41:25.316260 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 2128
I20260929 16:41:25.345489 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 2268
Sep 29 16:41:25 dist-test-slave-l31f krb5kdc[1872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700084, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.197.67@KRBTEST.COM
I20260929 16:41:25.364740 2422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:25.346310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50478 (local address 127.0.197.67:33555)
0929 16:41:25.346529 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:25.346546 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:25.346572 (+ 26us) server_negotiation.cc:407] Connection header received
0929 16:41:25.346673 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:25.346678 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:25.346774 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:25.346891 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:25.348224 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:25.349423 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:25.351051 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:25.351962 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:25.352105 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:25.357213 (+ 5108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:25.357235 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:25.357257 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:25.357310 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:25.361504 (+ 4194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:25.362400 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:25.362410 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:25.362414 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:25.362595 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:25.362998 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:25.363032 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:25.363036 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:25.363516 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:25.363897 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:25.364105 (+ 208us) server_negotiation.cc:299] Negotiation successful
0929 16:41:25.364345 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260929 16:41:25.365481 2399 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50478
W20260929 16:41:25.366199 789 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 2268: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:41:25.366385 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 2268
I20260929 16:41:25.385746 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 1885
I20260929 16:41:25.386899 1957 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:41:25.506460 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 1885
2026-09-29T16:41:25Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5037 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:41:25 dist-test-slave-l31f krb5kdc[2490](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:41:25 dist-test-slave-l31f krb5kdc[2490](info): set up 2 sockets
Sep 29 16:41:25 dist-test-slave-l31f krb5kdc[2490](info): commencing operation
krb5kdc: starting...
W20260929 16:41:26.627511 789 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/krb5kdc/test-user.keytab.
Sep 29 16:41:26 dist-test-slave-l31f krb5kdc[2490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:41:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:41:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.197.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
I20260929 16:41:26.835830 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:45257
--webserver_interface=127.0.197.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:43567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.126:45257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:41:27.187526 2503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:41:27.187844 2503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:41:27.188012 2503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:41:27.195711 2503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:41:27.195835 2503 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 16:41:27.195904 2503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:41:27.196005 2503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:41:27.196090 2503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:41:27.196208 2503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:41:27.207315 2503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:43567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.197.126:45257
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:45257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.197.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:41:27.208767 2503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:41:27.210203 2503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:41:27.223114 2509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:27.223064 2508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:27.223091 2511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:41:27.224503 2503 server_base.cc:1034] running on GCE node
I20260929 16:41:27.225406 2503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:41:27.227093 2503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:41:27.228339 2503 hybrid_clock.cc:648] HybridClock initialized: now 1790700087228280 us; error 40 us; skew 500 ppm
I20260929 16:41:27.228726 2503 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:41:27 dist-test-slave-l31f krb5kdc[2490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700087, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:41:27.232229 2503 init.cc:377] Logged in from keytab as kudu/127.0.197.126@KRBTEST.COM (short username kudu)
I20260929 16:41:27.234035 2503 webserver.cc:460] Webserver started at http://127.0.197.126:46431/ using document root <none> and password file <none>
I20260929 16:41:27.234870 2503 fs_manager.cc:359] Metadata directory not provided
I20260929 16:41:27.235110 2503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:41:27.235524 2503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:41:27.238904 2503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/master-0/data/instance:
uuid: "e8c03d81d4e8437d90edd6d0e53056a5"
format_stamp: "Formatted at 2026-09-29 16:41:27 on dist-test-slave-l31f"
server_key: "3fd718b4b316e2cfe5c9d682fa6f5b3c"
server_key_iv: "b29192fb3d24bcbd297bbd64a38a539e"
server_key_version: "encryptionkey@0"
I20260929 16:41:27.239912 2503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/master-0/wal/instance:
uuid: "e8c03d81d4e8437d90edd6d0e53056a5"
format_stamp: "Formatted at 2026-09-29 16:41:27 on dist-test-slave-l31f"
server_key: "3fd718b4b316e2cfe5c9d682fa6f5b3c"
server_key_iv: "b29192fb3d24bcbd297bbd64a38a539e"
server_key_version: "encryptionkey@0"
I20260929 16:41:27.245970 2503 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 16:41:27.250345 2518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:27.252198 2503 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260929 16:41:27.252540 2503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "e8c03d81d4e8437d90edd6d0e53056a5"
format_stamp: "Formatted at 2026-09-29 16:41:27 on dist-test-slave-l31f"
server_key: "3fd718b4b316e2cfe5c9d682fa6f5b3c"
server_key_iv: "b29192fb3d24bcbd297bbd64a38a539e"
server_key_version: "encryptionkey@0"
I20260929 16:41:27.252804 2503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:41:27.286794 2503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:41:27.291159 2503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:41:27.291456 2503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:41:27.314198 2503 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.126:45257
I20260929 16:41:27.314218 2579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.126:45257 every 8 connection(s)
I20260929 16:41:27.315874 2503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:41:27.317742 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 2503
I20260929 16:41:27.317975 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/master-0/wal/instance
I20260929 16:41:27.318619 789 external_mini_cluster.cc:1428] Setting key 15fd329e993cc8e5cfe3fca8d0457116
I20260929 16:41:27.320713 2580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 16:41:27 dist-test-slave-l31f krb5kdc[2490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:41:27.333840 2580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8c03d81d4e8437d90edd6d0e53056a5: Bootstrap starting.
I20260929 16:41:27.338274 2580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8c03d81d4e8437d90edd6d0e53056a5: Neither blocks nor log segments found. Creating new log.
I20260929 16:41:27.339789 2580 log.cc:824] T 00000000000000000000000000000000 P e8c03d81d4e8437d90edd6d0e53056a5: Log is configured to *not* fsync() on all Append() calls
I20260929 16:41:27.343500 2580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8c03d81d4e8437d90edd6d0e53056a5: No bootstrap required, opened a new log
I20260929 16:41:27.344619 2583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:27.321003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44978 (local address 127.0.197.126:45257)
0929 16:41:27.321851 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:27.321862 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:27.321905 (+ 43us) server_negotiation.cc:407] Connection header received
0929 16:41:27.323723 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:27.323748 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:27.324354 (+ 606us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:27.324822 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:27.326372 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:27.327819 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:27.329123 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:27.330272 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:27.330526 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:27.335117 (+ 4591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:27.335131 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:27.335154 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:27.335207 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:27.339443 (+ 4236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:27.340387 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:27.340402 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:27.340426 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:27.340722 (+ 296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:27.341157 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:27.341163 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:27.341166 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:27.341905 (+ 739us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:27.342201 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:27.342623 (+ 422us) server_negotiation.cc:299] Negotiation successful
0929 16:41:27.343121 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":439,"threads_started":1}
I20260929 16:41:27.352139 2580 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8c03d81d4e8437d90edd6d0e53056a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8c03d81d4e8437d90edd6d0e53056a5" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 45257 } }
I20260929 16:41:27.352617 2580 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8c03d81d4e8437d90edd6d0e53056a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:41:27.352732 2580 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8c03d81d4e8437d90edd6d0e53056a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8c03d81d4e8437d90edd6d0e53056a5, State: Initialized, Role: FOLLOWER
I20260929 16:41:27.353469 2580 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8c03d81d4e8437d90edd6d0e53056a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8c03d81d4e8437d90edd6d0e53056a5" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 45257 } }
I20260929 16:41:27.353786 2580 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8c03d81d4e8437d90edd6d0e53056a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:41:27.353953 2580 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8c03d81d4e8437d90edd6d0e53056a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:41:27.354157 2580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8c03d81d4e8437d90edd6d0e53056a5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:41:27.356604 2580 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8c03d81d4e8437d90edd6d0e53056a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8c03d81d4e8437d90edd6d0e53056a5" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 45257 } }
I20260929 16:41:27.357198 2580 leader_election.cc:302] T 00000000000000000000000000000000 P e8c03d81d4e8437d90edd6d0e53056a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8c03d81d4e8437d90edd6d0e53056a5; no voters:
I20260929 16:41:27.357880 2580 leader_election.cc:288] T 00000000000000000000000000000000 P e8c03d81d4e8437d90edd6d0e53056a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:41:27.358038 2585 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8c03d81d4e8437d90edd6d0e53056a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:41:27.359658 2585 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8c03d81d4e8437d90edd6d0e53056a5 [term 1 LEADER]: Becoming Leader. State: Replica: e8c03d81d4e8437d90edd6d0e53056a5, State: Running, Role: LEADER
I20260929 16:41:27.360378 2585 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8c03d81d4e8437d90edd6d0e53056a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8c03d81d4e8437d90edd6d0e53056a5" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 45257 } }
I20260929 16:41:27.360651 2580 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8c03d81d4e8437d90edd6d0e53056a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:41:27.368818 2587 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8c03d81d4e8437d90edd6d0e53056a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8c03d81d4e8437d90edd6d0e53056a5. Latest consensus state: current_term: 1 leader_uuid: "e8c03d81d4e8437d90edd6d0e53056a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8c03d81d4e8437d90edd6d0e53056a5" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 45257 } } }
I20260929 16:41:27.369210 2587 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8c03d81d4e8437d90edd6d0e53056a5 [sys.catalog]: This master's current role is: LEADER
I20260929 16:41:27.369979 2586 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8c03d81d4e8437d90edd6d0e53056a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8c03d81d4e8437d90edd6d0e53056a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8c03d81d4e8437d90edd6d0e53056a5" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 45257 } } }
I20260929 16:41:27.370256 2586 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8c03d81d4e8437d90edd6d0e53056a5 [sys.catalog]: This master's current role is: LEADER
I20260929 16:41:27.370358 2591 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:41:27.379885 2591 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:41:27.393234 2591 catalog_manager.cc:1347] Generated new cluster ID: df020812d4eb43cda72bff07aebcca3f
I20260929 16:41:27.393388 2591 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:41:27.417759 2591 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:41:27.419253 2591 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:41:27.435556 2591 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8c03d81d4e8437d90edd6d0e53056a5: Generated new TSK 0
I20260929 16:41:27.436765 2591 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.197.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab.
Entry for principal kudu/127.0.197.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab.
I20260929 16:41:27.496116 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.65:0
--local_ip_for_outbound_sockets=127.0.197.65
--webserver_interface=127.0.197.65
--webserver_port=0
--tserver_master_addrs=127.0.197.126:45257
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab
--principal=kudu/127.0.197.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:43567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:41:27.853518 2606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:41:27.853850 2606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:41:27.853971 2606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:41:27.861928 2606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:41:27.862087 2606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:41:27.862208 2606 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 16:41:27.862310 2606 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 16:41:27.862404 2606 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.65
I20260929 16:41:27.872917 2606 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:43567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab
--principal=kudu/127.0.197.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.197.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.197.126:45257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.65
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:41:27.874285 2606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:41:27.875648 2606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:41:27.890784 2611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:27.890919 2612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:27.890801 2614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:41:27.891664 2606 server_base.cc:1034] running on GCE node
I20260929 16:41:27.892370 2606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:41:27.893507 2606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:41:27.894790 2606 hybrid_clock.cc:648] HybridClock initialized: now 1790700087894717 us; error 71 us; skew 500 ppm
I20260929 16:41:27.895268 2606 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:41:27 dist-test-slave-l31f krb5kdc[2490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700087, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:41:27.898777 2606 init.cc:377] Logged in from keytab as kudu/127.0.197.65@KRBTEST.COM (short username kudu)
I20260929 16:41:27.900749 2606 webserver.cc:460] Webserver started at http://127.0.197.65:41895/ using document root <none> and password file <none>
I20260929 16:41:27.901592 2606 fs_manager.cc:359] Metadata directory not provided
I20260929 16:41:27.901757 2606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:41:27.902098 2606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:41:27.905467 2606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-0/data/instance:
uuid: "84d0189b4c7342218cf6f788923580d4"
format_stamp: "Formatted at 2026-09-29 16:41:27 on dist-test-slave-l31f"
server_key: "6cbd72d315f445abec427b410d8616a3"
server_key_iv: "dc3be7fa7a70047fd750f9381fe0d0ca"
server_key_version: "encryptionkey@0"
I20260929 16:41:27.906385 2606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-0/wal/instance:
uuid: "84d0189b4c7342218cf6f788923580d4"
format_stamp: "Formatted at 2026-09-29 16:41:27 on dist-test-slave-l31f"
server_key: "6cbd72d315f445abec427b410d8616a3"
server_key_iv: "dc3be7fa7a70047fd750f9381fe0d0ca"
server_key_version: "encryptionkey@0"
I20260929 16:41:27.912993 2606 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260929 16:41:27.917227 2621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:27.918955 2606 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:41:27.919293 2606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-0/wal
uuid: "84d0189b4c7342218cf6f788923580d4"
format_stamp: "Formatted at 2026-09-29 16:41:27 on dist-test-slave-l31f"
server_key: "6cbd72d315f445abec427b410d8616a3"
server_key_iv: "dc3be7fa7a70047fd750f9381fe0d0ca"
server_key_version: "encryptionkey@0"
I20260929 16:41:27.919538 2606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:41:27.942312 2606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:41:27.946960 2606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:41:27.947366 2606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:41:27.950122 2606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:41:27.950279 2606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:27.950441 2606 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:41:27.950536 2606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:41:27 dist-test-slave-l31f krb5kdc[2490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700087, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.65@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:41:27.977123 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:27.951177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:51515 (local address 127.0.197.126:45257)
0929 16:41:27.952289 (+ 1112us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:27.952300 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:27.953056 (+ 756us) server_negotiation.cc:407] Connection header received
0929 16:41:27.956391 (+ 3335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:27.956398 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:27.956501 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:27.956715 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:27.959479 (+ 2764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:27.960836 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:27.963651 (+ 2815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:27.964467 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:27.964655 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:27.969672 (+ 5017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:27.969678 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:27.969682 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:27.969720 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:27.972616 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:27.973896 (+ 1280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:27.973904 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:27.973907 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:27.974109 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:27.974745 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:27.974751 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:27.974755 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:27.975157 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:27.975336 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:27.976594 (+ 1258us) server_negotiation.cc:299] Negotiation successful
0929 16:41:27.976824 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":947,"thread_start_us":829,"threads_started":1}
I20260929 16:41:27.979430 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:27.951796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:45257 (local address 127.0.197.65:51515)
0929 16:41:27.952821 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:27.952893 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:27.955852 (+ 2959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:27.956993 (+ 1141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:27.957003 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:27.957898 (+ 895us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:41:27.958990 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:27.959072 (+ 82us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:27.961166 (+ 2094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:27.961180 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:27.963437 (+ 2257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:27.963454 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:27.964787 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:27.964798 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:27.965223 (+ 425us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:27.966168 (+ 945us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:41:27.966206 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:41:27.969263 (+ 3057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:41:27.972806 (+ 3543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:27.972817 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:27.972839 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:27.973672 (+ 833us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:27.974274 (+ 602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:27.974279 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:27.974284 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:27.974600 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:27.975326 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:41:27.975339 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:41:27.976030 (+ 691us) client_negotiation.cc:769] Sending connection context
0929 16:41:27.976454 (+ 424us) client_negotiation.cc:240] Negotiation successful
0929 16:41:27.977262 (+ 808us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":389,"threads_started":1}
I20260929 16:41:28.002945 2606 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.65:43273
I20260929 16:41:28.002962 2741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.65:43273 every 8 connection(s)
I20260929 16:41:28.004520 2606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
I20260929 16:41:28.005893 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:27.988093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:59361 (local address 127.0.197.126:45257)
0929 16:41:27.988349 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:27.988358 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:27.989069 (+ 711us) server_negotiation.cc:407] Connection header received
0929 16:41:27.989303 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:27.989311 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:27.989412 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:27.989541 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:27.991797 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:27.993010 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:27.995244 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:27.996102 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:27.996280 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:27.999682 (+ 3402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:27.999691 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:27.999699 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:27.999756 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:28.002845 (+ 3089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:28.003742 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:28.003747 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:28.003750 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:28.003896 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:28.004313 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:28.004318 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:28.004320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:28.004653 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:28.004843 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:28.005259 (+ 416us) server_negotiation.cc:299] Negotiation successful
0929 16:41:28.005512 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 16:41:28.006317 2707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:27.988063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:45257 (local address 127.0.197.65:59361)
0929 16:41:27.988914 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:27.988938 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:27.989146 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:27.989759 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:27.989772 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:27.990816 (+ 1044us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:41:27.991512 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:27.991534 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:27.993201 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:27.993205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:27.994995 (+ 1790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:27.995043 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:27.996367 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:27.996380 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:27.996640 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:27.997624 (+ 984us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:41:27.997658 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:41:27.999339 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:41:28.003061 (+ 3722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:28.003073 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:28.003077 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:28.003575 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:28.004023 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:28.004031 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:28.004035 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:28.004180 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:28.004782 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:41:28.004787 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:41:28.004998 (+ 211us) client_negotiation.cc:769] Sending connection context
0929 16:41:28.005155 (+ 157us) client_negotiation.cc:240] Negotiation successful
0929 16:41:28.005540 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":444,"threads_started":1}
I20260929 16:41:28.008019 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 2606
I20260929 16:41:28.007964 2742 heartbeater.cc:347] Connected to a master server at 127.0.197.126:45257
I20260929 16:41:28.008219 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-0/wal/instance
I20260929 16:41:28.008376 2742 heartbeater.cc:464] Registering TS with master...
I20260929 16:41:28.009057 789 external_mini_cluster.cc:1428] Setting key 469758f93fde6f81c668516b27ac3c89
I20260929 16:41:28.009567 2742 heartbeater.cc:511] Master 127.0.197.126:45257 requested a full tablet report, sending...
I20260929 16:41:28.013392 2534 ts_manager.cc:194] Registered new tserver with Master: 84d0189b4c7342218cf6f788923580d4 (127.0.197.65:43273)
I20260929 16:41:28.015262 2534 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.65@KRBTEST.COM'} at 127.0.197.65:51515
WARNING: no policy specified for kudu/127.0.197.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab.
Entry for principal kudu/127.0.197.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab.
I20260929 16:41:28.059818 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.66:0
--local_ip_for_outbound_sockets=127.0.197.66
--webserver_interface=127.0.197.66
--webserver_port=0
--tserver_master_addrs=127.0.197.126:45257
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab
--principal=kudu/127.0.197.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:43567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:41:28.421888 2746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:41:28.422206 2746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:41:28.422379 2746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:41:28.430343 2746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:41:28.430486 2746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:41:28.430644 2746 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 16:41:28.430706 2746 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 16:41:28.430824 2746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.66
I20260929 16:41:28.441519 2746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:43567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab
--principal=kudu/127.0.197.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.197.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.197.126:45257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.66
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:41:28.443085 2746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:41:28.444599 2746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:41:28.459532 2754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:28.459551 2752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:28.459601 2751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:41:28.460495 2746 server_base.cc:1034] running on GCE node
I20260929 16:41:28.461127 2746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:41:28.462288 2746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:41:28.463554 2746 hybrid_clock.cc:648] HybridClock initialized: now 1790700088463494 us; error 79 us; skew 500 ppm
I20260929 16:41:28.463888 2746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:41:28 dist-test-slave-l31f krb5kdc[2490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700088, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:41:28.467473 2746 init.cc:377] Logged in from keytab as kudu/127.0.197.66@KRBTEST.COM (short username kudu)
I20260929 16:41:28.469168 2746 webserver.cc:460] Webserver started at http://127.0.197.66:42467/ using document root <none> and password file <none>
I20260929 16:41:28.470021 2746 fs_manager.cc:359] Metadata directory not provided
I20260929 16:41:28.470141 2746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:41:28.470428 2746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:41:28.473654 2746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-1/data/instance:
uuid: "45f0267549da47abb6a5ee35c3509a9a"
format_stamp: "Formatted at 2026-09-29 16:41:28 on dist-test-slave-l31f"
server_key: "065081661e771376b38bc7a14cbda92a"
server_key_iv: "b6653da02f524a1a18c37569be65288f"
server_key_version: "encryptionkey@0"
I20260929 16:41:28.474490 2746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-1/wal/instance:
uuid: "45f0267549da47abb6a5ee35c3509a9a"
format_stamp: "Formatted at 2026-09-29 16:41:28 on dist-test-slave-l31f"
server_key: "065081661e771376b38bc7a14cbda92a"
server_key_iv: "b6653da02f524a1a18c37569be65288f"
server_key_version: "encryptionkey@0"
I20260929 16:41:28.480400 2746 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 16:41:28.484579 2761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:28.486306 2746 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:41:28.486536 2746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-1/wal
uuid: "45f0267549da47abb6a5ee35c3509a9a"
format_stamp: "Formatted at 2026-09-29 16:41:28 on dist-test-slave-l31f"
server_key: "065081661e771376b38bc7a14cbda92a"
server_key_iv: "b6653da02f524a1a18c37569be65288f"
server_key_version: "encryptionkey@0"
I20260929 16:41:28.486769 2746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:41:28.522404 2746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:41:28.527374 2746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:41:28.527709 2746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:41:28.530591 2746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:41:28.530755 2746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:28.530900 2746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:41:28.530994 2746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:41:28 dist-test-slave-l31f krb5kdc[2490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700088, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.66@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:41:28.557417 2770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:28.531506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:58795 (local address 127.0.197.126:45257)
0929 16:41:28.532030 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:28.532038 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:28.533505 (+ 1467us) server_negotiation.cc:407] Connection header received
0929 16:41:28.537045 (+ 3540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:28.537052 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:28.537147 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:28.537272 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:28.540054 (+ 2782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:28.540837 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:28.542606 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:28.543126 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:28.543257 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:28.548148 (+ 4891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:28.548160 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:28.548168 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:28.548231 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:28.552559 (+ 4328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:28.554084 (+ 1525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:28.554092 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:28.554098 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:28.554310 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:28.554889 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:28.554896 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:28.554902 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:28.555483 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:28.555732 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:28.556730 (+ 998us) server_negotiation.cc:299] Negotiation successful
0929 16:41:28.557012 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":233,"threads_started":1}
I20260929 16:41:28.558550 2771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:28.532326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:45257 (local address 127.0.197.66:58795)
0929 16:41:28.533277 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:28.533336 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:28.536575 (+ 3239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:28.537681 (+ 1106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:28.537699 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:28.538687 (+ 988us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:41:28.539804 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:28.539829 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:28.541019 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:28.541025 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:28.542431 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:28.542445 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:28.543442 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:28.543450 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:28.543653 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:28.544624 (+ 971us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:41:28.544666 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:41:28.547789 (+ 3123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:41:28.552812 (+ 5023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:28.552826 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:28.552856 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:28.553880 (+ 1024us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:28.554484 (+ 604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:28.554488 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:28.554490 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:28.554729 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:28.555724 (+ 995us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:41:28.555740 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:41:28.556273 (+ 533us) client_negotiation.cc:769] Sending connection context
0929 16:41:28.556610 (+ 337us) client_negotiation.cc:240] Negotiation successful
0929 16:41:28.557252 (+ 642us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":402,"threads_started":1}
I20260929 16:41:28.587074 2834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:28.570499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:45257 (local address 127.0.197.66:60229)
0929 16:41:28.571464 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:28.571487 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:28.571709 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:28.572104 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:28.572108 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:28.572673 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:41:28.573008 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:28.573020 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:28.574425 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:28.574434 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:28.575771 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:28.575788 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:28.577191 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:28.577208 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:28.577390 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:28.578182 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:41:28.578213 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:41:28.579230 (+ 1017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:41:28.583570 (+ 4340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:28.583576 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:28.583581 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:28.584301 (+ 720us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:28.584756 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:28.584762 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:28.584765 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:28.584924 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:28.585785 (+ 861us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:41:28.585788 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:41:28.586049 (+ 261us) client_negotiation.cc:769] Sending connection context
0929 16:41:28.586243 (+ 194us) client_negotiation.cc:240] Negotiation successful
0929 16:41:28.586631 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"thread_start_us":565,"threads_started":1}
I20260929 16:41:28.587510 2770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:28.570371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:60229 (local address 127.0.197.126:45257)
0929 16:41:28.570616 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:28.570626 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:28.571622 (+ 996us) server_negotiation.cc:407] Connection header received
0929 16:41:28.571872 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:28.571878 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:28.571957 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:28.572072 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:28.573179 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:28.574277 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:28.575958 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:28.576617 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:28.576963 (+ 346us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:28.579482 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:28.579488 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:28.579492 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:28.579527 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:28.583315 (+ 3788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:28.584469 (+ 1154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:28.584475 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:28.584479 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:28.584623 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:28.585064 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:28.585069 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:28.585073 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:28.585561 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:28.585718 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:28.586602 (+ 884us) server_negotiation.cc:299] Negotiation successful
0929 16:41:28.586963 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 16:41:28.590768 2746 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.66:43603
I20260929 16:41:28.590838 2881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.66:43603 every 8 connection(s)
I20260929 16:41:28.592401 2746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
I20260929 16:41:28.594738 2882 heartbeater.cc:347] Connected to a master server at 127.0.197.126:45257
I20260929 16:41:28.595057 2882 heartbeater.cc:464] Registering TS with master...
I20260929 16:41:28.595872 2882 heartbeater.cc:511] Master 127.0.197.126:45257 requested a full tablet report, sending...
I20260929 16:41:28.597405 2534 ts_manager.cc:194] Registered new tserver with Master: 45f0267549da47abb6a5ee35c3509a9a (127.0.197.66:43603)
I20260929 16:41:28.598217 2534 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.66@KRBTEST.COM'} at 127.0.197.66:58795
I20260929 16:41:28.602409 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 2746
I20260929 16:41:28.602553 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-1/wal/instance
I20260929 16:41:28.602936 789 external_mini_cluster.cc:1428] Setting key 2c7aab4c345d395c99a1ed8b66978300
WARNING: no policy specified for kudu/127.0.197.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab.
Entry for principal kudu/127.0.197.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab.
I20260929 16:41:28.651224 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.67:0
--local_ip_for_outbound_sockets=127.0.197.67
--webserver_interface=127.0.197.67
--webserver_port=0
--tserver_master_addrs=127.0.197.126:45257
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab
--principal=kudu/127.0.197.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:43567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:41:29.008925 2886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:41:29.009268 2886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:41:29.009447 2886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:41:29.017808 2886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:41:29.018018 2886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:41:29.018190 2886 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 16:41:29.018276 2886 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 16:41:29.018385 2886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.67
I20260929 16:41:29.019373 2742 heartbeater.cc:503] Master 127.0.197.126:45257 was elected leader, sending a full tablet report...
I20260929 16:41:29.029392 2886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:43567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab
--principal=kudu/127.0.197.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.197.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.197.126:45257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.67
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:41:29.030951 2886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:41:29.033334 2886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:41:29.049265 2891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:29.049232 2892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:41:29.049705 2886 server_base.cc:1034] running on GCE node
W20260929 16:41:29.049192 2894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:41:29.050777 2886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:41:29.051877 2886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:41:29.053125 2886 hybrid_clock.cc:648] HybridClock initialized: now 1790700089053074 us; error 40 us; skew 500 ppm
I20260929 16:41:29.053515 2886 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:41:29 dist-test-slave-l31f krb5kdc[2490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700089, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:41:29.057097 2886 init.cc:377] Logged in from keytab as kudu/127.0.197.67@KRBTEST.COM (short username kudu)
I20260929 16:41:29.058907 2886 webserver.cc:460] Webserver started at http://127.0.197.67:44491/ using document root <none> and password file <none>
I20260929 16:41:29.059711 2886 fs_manager.cc:359] Metadata directory not provided
I20260929 16:41:29.059875 2886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:41:29.060276 2886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:41:29.063939 2886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-2/data/instance:
uuid: "c8d5e1e737d147d5b6d47fe948fc3623"
format_stamp: "Formatted at 2026-09-29 16:41:29 on dist-test-slave-l31f"
server_key: "50e56e9c36622b51065a07e825ab7446"
server_key_iv: "6b00de5b6004c18b6c863d2fd94c0b8f"
server_key_version: "encryptionkey@0"
I20260929 16:41:29.064785 2886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-2/wal/instance:
uuid: "c8d5e1e737d147d5b6d47fe948fc3623"
format_stamp: "Formatted at 2026-09-29 16:41:29 on dist-test-slave-l31f"
server_key: "50e56e9c36622b51065a07e825ab7446"
server_key_iv: "6b00de5b6004c18b6c863d2fd94c0b8f"
server_key_version: "encryptionkey@0"
I20260929 16:41:29.070798 2886 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 16:41:29.074949 2901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:29.076617 2886 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:41:29.076858 2886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-2/wal
uuid: "c8d5e1e737d147d5b6d47fe948fc3623"
format_stamp: "Formatted at 2026-09-29 16:41:29 on dist-test-slave-l31f"
server_key: "50e56e9c36622b51065a07e825ab7446"
server_key_iv: "6b00de5b6004c18b6c863d2fd94c0b8f"
server_key_version: "encryptionkey@0"
I20260929 16:41:29.077072 2886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:41:29.092967 2886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:41:29.097796 2886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:41:29.098171 2886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:41:29.101236 2886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:41:29.101392 2886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:29.101541 2886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:41:29.101676 2886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 29 16:41:29 dist-test-slave-l31f krb5kdc[2490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700089, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.67@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:41:29.131867 2909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:29.101715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:33083 (local address 127.0.197.126:45257)
0929 16:41:29.102228 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:29.102236 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:29.103700 (+ 1464us) server_negotiation.cc:407] Connection header received
0929 16:41:29.106034 (+ 2334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:29.106043 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:29.106152 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:29.106292 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:29.110535 (+ 4243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:29.111641 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:29.114052 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:29.114701 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:29.114897 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:29.121538 (+ 6641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:29.121549 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:29.121558 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:29.121617 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:29.126244 (+ 4627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:29.128136 (+ 1892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:29.128144 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:29.128151 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:29.128369 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:29.129054 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:29.129063 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:29.129069 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:29.129498 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:29.129668 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:29.130948 (+ 1280us) server_negotiation.cc:299] Negotiation successful
0929 16:41:29.131321 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":240,"threads_started":1}
I20260929 16:41:29.133698 2911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:29.102533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:45257 (local address 127.0.197.67:33083)
0929 16:41:29.103511 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:29.103568 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:29.105536 (+ 1968us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:29.106697 (+ 1161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:29.106712 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:29.108986 (+ 2274us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:41:29.110215 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:29.110264 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:29.111849 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:29.111854 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:29.113833 (+ 1979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:29.113850 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:29.115062 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:29.115072 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:29.115360 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:29.116602 (+ 1242us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:41:29.116655 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:41:29.121087 (+ 4432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:41:29.126516 (+ 5429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:29.126532 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:29.126566 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:29.127916 (+ 1350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:29.128505 (+ 589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:29.128513 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:29.128518 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:29.128886 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:29.129677 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:41:29.129689 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:41:29.130356 (+ 667us) client_negotiation.cc:769] Sending connection context
0929 16:41:29.130771 (+ 415us) client_negotiation.cc:240] Negotiation successful
0929 16:41:29.131587 (+ 816us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":712,"thread_start_us":500,"threads_started":1}
I20260929 16:41:29.147543 2886 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.67:46267
I20260929 16:41:29.147642 3021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.67:46267 every 8 connection(s)
I20260929 16:41:29.149040 2886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
I20260929 16:41:29.151473 3022 heartbeater.cc:347] Connected to a master server at 127.0.197.126:45257
I20260929 16:41:29.151752 3022 heartbeater.cc:464] Registering TS with master...
I20260929 16:41:29.152599 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 2886
I20260929 16:41:29.152617 3022 heartbeater.cc:511] Master 127.0.197.126:45257 requested a full tablet report, sending...
I20260929 16:41:29.152791 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-2/wal/instance
I20260929 16:41:29.153290 789 external_mini_cluster.cc:1428] Setting key 7acf44b61c48017b2c702dc20f815e6c
I20260929 16:41:29.154709 2534 ts_manager.cc:194] Registered new tserver with Master: c8d5e1e737d147d5b6d47fe948fc3623 (127.0.197.67:46267)
I20260929 16:41:29.156018 2534 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.67@KRBTEST.COM'} at 127.0.197.67:33083
I20260929 16:41:29.156610 789 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:41:29.158334 2909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:29.141894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:40129 (local address 127.0.197.126:45257)
0929 16:41:29.142165 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:29.142175 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:29.143287 (+ 1112us) server_negotiation.cc:407] Connection header received
0929 16:41:29.143578 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:29.143584 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:29.143668 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:29.143787 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:29.145367 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:29.146107 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:29.148310 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:29.148769 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:29.148950 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:29.151515 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:29.151522 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:29.151527 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:29.151571 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:29.155168 (+ 3597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:29.156049 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:29.156056 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:29.156059 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:29.156230 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:29.156712 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:29.156717 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:29.156720 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:29.157079 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:29.157248 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:29.157729 (+ 481us) server_negotiation.cc:299] Negotiation successful
0929 16:41:29.157994 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 16:41:29.158433 3010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:29.142255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:45257 (local address 127.0.197.67:40129)
0929 16:41:29.143141 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:29.143173 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:29.143401 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:29.143844 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:29.143850 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:29.144628 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:41:29.145153 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:29.145177 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:29.146292 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:29.146297 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:29.148131 (+ 1834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:29.148146 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:29.149002 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:29.149009 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:29.149249 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:29.150124 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:41:29.150146 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:41:29.151295 (+ 1149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:41:29.155318 (+ 4023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:29.155323 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:29.155329 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:29.155857 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:29.156402 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:29.156409 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:29.156412 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:29.156580 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:29.157205 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:41:29.157210 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:41:29.157428 (+ 218us) client_negotiation.cc:769] Sending connection context
0929 16:41:29.157592 (+ 164us) client_negotiation.cc:240] Negotiation successful
0929 16:41:29.157946 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"thread_start_us":465,"threads_started":1}
Sep 29 16:41:29 dist-test-slave-l31f krb5kdc[2490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 16:41:29 dist-test-slave-l31f krb5kdc[2490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:41:29.208583 2909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:29.193534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44982 (local address 127.0.197.126:45257)
0929 16:41:29.193778 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:29.193784 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:29.194020 (+ 236us) server_negotiation.cc:407] Connection header received
0929 16:41:29.194173 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:29.194177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:29.194242 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:29.194334 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:29.195889 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:29.196648 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:29.197960 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:29.198372 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:29.198482 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:29.202950 (+ 4468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:29.202956 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:29.202960 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:29.203057 (+ 97us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:29.205570 (+ 2513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:29.206469 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:29.206475 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:29.206478 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:29.206733 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:29.207270 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:29.207276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:29.207279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:29.207637 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:29.207862 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:29.208143 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 16:41:29.208322 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 16:41:29.213132 2534 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44982:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:41:29.220170 2534 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:41:29.246603 3037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:29.235000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40104 (local address 127.0.197.67:46267)
0929 16:41:29.236233 (+ 1233us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:29.236247 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:29.237511 (+ 1264us) server_negotiation.cc:407] Connection header received
0929 16:41:29.238013 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:29.238020 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:29.238292 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:29.238473 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:41:29.240057 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:29.241729 (+ 1672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:29.244228 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:29.245249 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:29.245438 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:29.245711 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:29.246050 (+ 339us) server_negotiation.cc:299] Negotiation successful
0929 16:41:29.246245 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":968,"thread_start_us":710,"threads_started":1}
I20260929 16:41:29.246641 3036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:29.234317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.67:46267 (local address 127.0.0.1:40104)
0929 16:41:29.237335 (+ 3018us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:29.237395 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:29.237704 (+ 309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:29.238783 (+ 1079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:29.238791 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:29.238835 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:41:29.239856 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:29.239864 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:29.241922 (+ 2058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:29.241938 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:29.244068 (+ 2130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:29.244081 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:29.245503 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:29.245509 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:29.245709 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:29.245781 (+ 72us) client_negotiation.cc:769] Sending connection context
0929 16:41:29.245924 (+ 143us) client_negotiation.cc:240] Negotiation successful
0929 16:41:29.246156 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2668,"mutex_wait_us":686,"spinlock_wait_cycles":9344,"thread_start_us":2471,"threads_started":1}
I20260929 16:41:29.247095 3035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:29.234146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59056 (local address 127.0.197.65:43273)
0929 16:41:29.236681 (+ 2535us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:29.236690 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:29.237724 (+ 1034us) server_negotiation.cc:407] Connection header received
0929 16:41:29.238009 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:29.238018 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:29.238292 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:29.238473 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:41:29.239568 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:29.240907 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:29.243871 (+ 2964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:29.245445 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:29.245626 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:29.245827 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:29.246300 (+ 473us) server_negotiation.cc:299] Negotiation successful
0929 16:41:29.246491 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2387,"thread_start_us":2247,"threads_started":1}
I20260929 16:41:29.247699 3038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:29.235423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:43273 (local address 127.0.0.1:59056)
0929 16:41:29.237335 (+ 1912us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:29.237636 (+ 301us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:29.237815 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:29.238787 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:29.238791 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:29.238833 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:41:29.239372 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:29.239386 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:29.241052 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:29.241057 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:29.243643 (+ 2586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:29.243665 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:29.245878 (+ 2213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:29.245887 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:29.246093 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:29.246123 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 16:41:29.247148 (+ 1025us) client_negotiation.cc:240] Negotiation successful
0929 16:41:29.247311 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1634,"mutex_wait_us":10,"spinlock_wait_cycles":24832,"thread_start_us":1417,"threads_started":1}
I20260929 16:41:29.250849 2952 tablet_service.cc:1467] Processing CreateTablet for tablet 10e06387b28546a8b69e7c2d9168fee4 (DEFAULT_TABLE table=test-table [id=887b75c324d140b0b1aae84b76632a10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:41:29.251052 2672 tablet_service.cc:1467] Processing CreateTablet for tablet 10e06387b28546a8b69e7c2d9168fee4 (DEFAULT_TABLE table=test-table [id=887b75c324d140b0b1aae84b76632a10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:41:29.252511 3040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:29.238291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43720 (local address 127.0.197.66:43603)
0929 16:41:29.239178 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:29.239186 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:29.239532 (+ 346us) server_negotiation.cc:407] Connection header received
0929 16:41:29.239683 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:29.239692 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:29.239954 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:29.240165 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:41:29.240755 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:29.243157 (+ 2402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:29.245656 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:29.248374 (+ 2718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:29.248561 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:29.248783 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:29.251990 (+ 3207us) server_negotiation.cc:299] Negotiation successful
0929 16:41:29.252192 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":733,"thread_start_us":601,"threads_started":1}
I20260929 16:41:29.251816 3039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:29.237822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.66:43603 (local address 127.0.0.1:43720)
0929 16:41:29.239372 (+ 1550us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:29.239392 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:29.239527 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:29.240166 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:29.240171 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:29.240201 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:41:29.240564 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:29.240575 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:29.243303 (+ 2728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:29.243309 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:29.245441 (+ 2132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:29.245456 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:29.248612 (+ 3156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:29.248618 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:29.251096 (+ 2478us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:29.251141 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 16:41:29.251278 (+ 137us) client_negotiation.cc:240] Negotiation successful
0929 16:41:29.251438 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1401,"thread_start_us":1276,"threads_started":1}
I20260929 16:41:29.252619 2672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10e06387b28546a8b69e7c2d9168fee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:41:29.252892 2952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10e06387b28546a8b69e7c2d9168fee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:41:29.256529 2812 tablet_service.cc:1467] Processing CreateTablet for tablet 10e06387b28546a8b69e7c2d9168fee4 (DEFAULT_TABLE table=test-table [id=887b75c324d140b0b1aae84b76632a10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:41:29.258070 2812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10e06387b28546a8b69e7c2d9168fee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:41:29.267444 3041 tablet_bootstrap.cc:492] T 10e06387b28546a8b69e7c2d9168fee4 P 84d0189b4c7342218cf6f788923580d4: Bootstrap starting.
I20260929 16:41:29.270902 3042 tablet_bootstrap.cc:492] T 10e06387b28546a8b69e7c2d9168fee4 P c8d5e1e737d147d5b6d47fe948fc3623: Bootstrap starting.
I20260929 16:41:29.270987 3041 tablet_bootstrap.cc:654] T 10e06387b28546a8b69e7c2d9168fee4 P 84d0189b4c7342218cf6f788923580d4: Neither blocks nor log segments found. Creating new log.
I20260929 16:41:29.272578 3041 log.cc:824] T 10e06387b28546a8b69e7c2d9168fee4 P 84d0189b4c7342218cf6f788923580d4: Log is configured to *not* fsync() on all Append() calls
I20260929 16:41:29.275106 3044 tablet_bootstrap.cc:492] T 10e06387b28546a8b69e7c2d9168fee4 P 45f0267549da47abb6a5ee35c3509a9a: Bootstrap starting.
I20260929 16:41:29.275511 3042 tablet_bootstrap.cc:654] T 10e06387b28546a8b69e7c2d9168fee4 P c8d5e1e737d147d5b6d47fe948fc3623: Neither blocks nor log segments found. Creating new log.
I20260929 16:41:29.276150 3041 tablet_bootstrap.cc:492] T 10e06387b28546a8b69e7c2d9168fee4 P 84d0189b4c7342218cf6f788923580d4: No bootstrap required, opened a new log
I20260929 16:41:29.276469 3041 ts_tablet_manager.cc:1397] T 10e06387b28546a8b69e7c2d9168fee4 P 84d0189b4c7342218cf6f788923580d4: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20260929 16:41:29.276844 3042 log.cc:824] T 10e06387b28546a8b69e7c2d9168fee4 P c8d5e1e737d147d5b6d47fe948fc3623: Log is configured to *not* fsync() on all Append() calls
I20260929 16:41:29.278934 3044 tablet_bootstrap.cc:654] T 10e06387b28546a8b69e7c2d9168fee4 P 45f0267549da47abb6a5ee35c3509a9a: Neither blocks nor log segments found. Creating new log.
I20260929 16:41:29.280107 3044 log.cc:824] T 10e06387b28546a8b69e7c2d9168fee4 P 45f0267549da47abb6a5ee35c3509a9a: Log is configured to *not* fsync() on all Append() calls
I20260929 16:41:29.280366 3042 tablet_bootstrap.cc:492] T 10e06387b28546a8b69e7c2d9168fee4 P c8d5e1e737d147d5b6d47fe948fc3623: No bootstrap required, opened a new log
I20260929 16:41:29.280694 3042 ts_tablet_manager.cc:1397] T 10e06387b28546a8b69e7c2d9168fee4 P c8d5e1e737d147d5b6d47fe948fc3623: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260929 16:41:29.283542 3044 tablet_bootstrap.cc:492] T 10e06387b28546a8b69e7c2d9168fee4 P 45f0267549da47abb6a5ee35c3509a9a: No bootstrap required, opened a new log
I20260929 16:41:29.283875 3044 ts_tablet_manager.cc:1397] T 10e06387b28546a8b69e7c2d9168fee4 P 45f0267549da47abb6a5ee35c3509a9a: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260929 16:41:29.287392 3041 raft_consensus.cc:348] T 10e06387b28546a8b69e7c2d9168fee4 P 84d0189b4c7342218cf6f788923580d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } }
I20260929 16:41:29.287914 3041 raft_consensus.cc:374] T 10e06387b28546a8b69e7c2d9168fee4 P 84d0189b4c7342218cf6f788923580d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:41:29.288028 3041 raft_consensus.cc:729] T 10e06387b28546a8b69e7c2d9168fee4 P 84d0189b4c7342218cf6f788923580d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84d0189b4c7342218cf6f788923580d4, State: Initialized, Role: FOLLOWER
I20260929 16:41:29.288734 3041 consensus_queue.cc:260] T 10e06387b28546a8b69e7c2d9168fee4 P 84d0189b4c7342218cf6f788923580d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } }
I20260929 16:41:29.292279 3041 ts_tablet_manager.cc:1428] T 10e06387b28546a8b69e7c2d9168fee4 P 84d0189b4c7342218cf6f788923580d4: Time spent starting tablet: real 0.016s user 0.013s sys 0.001s
I20260929 16:41:29.292100 3042 raft_consensus.cc:348] T 10e06387b28546a8b69e7c2d9168fee4 P c8d5e1e737d147d5b6d47fe948fc3623 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } }
I20260929 16:41:29.292593 3042 raft_consensus.cc:374] T 10e06387b28546a8b69e7c2d9168fee4 P c8d5e1e737d147d5b6d47fe948fc3623 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:41:29.292698 3042 raft_consensus.cc:729] T 10e06387b28546a8b69e7c2d9168fee4 P c8d5e1e737d147d5b6d47fe948fc3623 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d5e1e737d147d5b6d47fe948fc3623, State: Initialized, Role: FOLLOWER
I20260929 16:41:29.293354 3042 consensus_queue.cc:260] T 10e06387b28546a8b69e7c2d9168fee4 P c8d5e1e737d147d5b6d47fe948fc3623 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } }
I20260929 16:41:29.293824 3044 raft_consensus.cc:348] T 10e06387b28546a8b69e7c2d9168fee4 P 45f0267549da47abb6a5ee35c3509a9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } }
I20260929 16:41:29.294312 3044 raft_consensus.cc:374] T 10e06387b28546a8b69e7c2d9168fee4 P 45f0267549da47abb6a5ee35c3509a9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:41:29.294425 3044 raft_consensus.cc:729] T 10e06387b28546a8b69e7c2d9168fee4 P 45f0267549da47abb6a5ee35c3509a9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45f0267549da47abb6a5ee35c3509a9a, State: Initialized, Role: FOLLOWER
I20260929 16:41:29.294559 3022 heartbeater.cc:503] Master 127.0.197.126:45257 was elected leader, sending a full tablet report...
I20260929 16:41:29.295254 3044 consensus_queue.cc:260] T 10e06387b28546a8b69e7c2d9168fee4 P 45f0267549da47abb6a5ee35c3509a9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } }
I20260929 16:41:29.296314 2882 heartbeater.cc:503] Master 127.0.197.126:45257 was elected leader, sending a full tablet report...
I20260929 16:41:29.298091 3042 ts_tablet_manager.cc:1428] T 10e06387b28546a8b69e7c2d9168fee4 P c8d5e1e737d147d5b6d47fe948fc3623: Time spent starting tablet: real 0.017s user 0.014s sys 0.002s
I20260929 16:41:29.298306 3044 ts_tablet_manager.cc:1428] T 10e06387b28546a8b69e7c2d9168fee4 P 45f0267549da47abb6a5ee35c3509a9a: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
W20260929 16:41:29.344617 2883 tablet.cc:2369] T 10e06387b28546a8b69e7c2d9168fee4 P 45f0267549da47abb6a5ee35c3509a9a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:41:29.401602 3023 tablet.cc:2369] T 10e06387b28546a8b69e7c2d9168fee4 P c8d5e1e737d147d5b6d47fe948fc3623: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:41:29.513577 2743 tablet.cc:2369] T 10e06387b28546a8b69e7c2d9168fee4 P 84d0189b4c7342218cf6f788923580d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:41:29.569010 3049 raft_consensus.cc:482] T 10e06387b28546a8b69e7c2d9168fee4 P 45f0267549da47abb6a5ee35c3509a9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:41:29.569391 3049 raft_consensus.cc:504] T 10e06387b28546a8b69e7c2d9168fee4 P 45f0267549da47abb6a5ee35c3509a9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } }
I20260929 16:41:29.571434 3049 leader_election.cc:288] T 10e06387b28546a8b69e7c2d9168fee4 P 45f0267549da47abb6a5ee35c3509a9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84d0189b4c7342218cf6f788923580d4 (127.0.197.65:43273), c8d5e1e737d147d5b6d47fe948fc3623 (127.0.197.67:46267)
I20260929 16:41:29.581370 3035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:29.572113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:50347 (local address 127.0.197.65:43273)
0929 16:41:29.572326 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:29.572334 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:29.573065 (+ 731us) server_negotiation.cc:407] Connection header received
0929 16:41:29.573352 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:29.573358 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:29.573455 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:29.573565 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:41:29.574429 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:29.575561 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:29.578726 (+ 3165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:29.580038 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:29.580231 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:29.580312 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:29.580966 (+ 654us) server_negotiation.cc:299] Negotiation successful
0929 16:41:29.581108 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 16:41:29.581385 3037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:29.572022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:48277 (local address 127.0.197.67:46267)
0929 16:41:29.572247 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:29.572256 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:29.573279 (+ 1023us) server_negotiation.cc:407] Connection header received
0929 16:41:29.573389 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:29.573393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:29.573461 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:29.573569 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:41:29.574409 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:29.575569 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:29.578715 (+ 3146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:29.580038 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:29.580231 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:29.580312 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:29.580966 (+ 654us) server_negotiation.cc:299] Negotiation successful
0929 16:41:29.581108 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 16:41:29.581641 3051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:29.572076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.67:46267 (local address 127.0.197.66:48277)
0929 16:41:29.573116 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:29.573138 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:29.573258 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:29.573636 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:29.573642 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:29.573674 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:41:29.574150 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:29.574170 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:29.575738 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:29.575745 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:29.578466 (+ 2721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:29.578486 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:29.580422 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:29.580431 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:29.580700 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:29.580778 (+ 78us) client_negotiation.cc:769] Sending connection context
0929 16:41:29.580918 (+ 140us) client_negotiation.cc:240] Negotiation successful
0929 16:41:29.581067 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":898,"mutex_wait_us":226,"thread_start_us":710,"threads_started":1}
I20260929 16:41:29.582077 3050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:29.571796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:43273 (local address 127.0.197.66:50347)
0929 16:41:29.572924 (+ 1128us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:29.572955 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:29.573146 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:29.573636 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:29.573642 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:29.573739 (+ 97us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:41:29.574239 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:29.574266 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:29.575738 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:29.575745 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:29.578464 (+ 2719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:29.578486 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:29.580429 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:29.580434 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:29.580687 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:29.580754 (+ 67us) client_negotiation.cc:769] Sending connection context
0929 16:41:29.580876 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 16:41:29.581067 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":917,"mutex_wait_us":171,"spinlock_wait_cycles":25344,"thread_start_us":638,"threads_started":1}
I20260929 16:41:29.582801 2972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e06387b28546a8b69e7c2d9168fee4" candidate_uuid: "45f0267549da47abb6a5ee35c3509a9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" is_pre_election: true
I20260929 16:41:29.583097 2692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e06387b28546a8b69e7c2d9168fee4" candidate_uuid: "45f0267549da47abb6a5ee35c3509a9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d0189b4c7342218cf6f788923580d4" is_pre_election: true
I20260929 16:41:29.583474 2972 raft_consensus.cc:2454] T 10e06387b28546a8b69e7c2d9168fee4 P c8d5e1e737d147d5b6d47fe948fc3623 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45f0267549da47abb6a5ee35c3509a9a in term 0.
I20260929 16:41:29.583547 2692 raft_consensus.cc:2454] T 10e06387b28546a8b69e7c2d9168fee4 P 84d0189b4c7342218cf6f788923580d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45f0267549da47abb6a5ee35c3509a9a in term 0.
I20260929 16:41:29.584314 2763 leader_election.cc:302] T 10e06387b28546a8b69e7c2d9168fee4 P 45f0267549da47abb6a5ee35c3509a9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 45f0267549da47abb6a5ee35c3509a9a, 84d0189b4c7342218cf6f788923580d4, c8d5e1e737d147d5b6d47fe948fc3623; no voters:
I20260929 16:41:29.584839 3049 raft_consensus.cc:2781] T 10e06387b28546a8b69e7c2d9168fee4 P 45f0267549da47abb6a5ee35c3509a9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:41:29.585039 3049 raft_consensus.cc:482] T 10e06387b28546a8b69e7c2d9168fee4 P 45f0267549da47abb6a5ee35c3509a9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:41:29.585178 3049 raft_consensus.cc:3037] T 10e06387b28546a8b69e7c2d9168fee4 P 45f0267549da47abb6a5ee35c3509a9a [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:41:29.587770 3049 raft_consensus.cc:504] T 10e06387b28546a8b69e7c2d9168fee4 P 45f0267549da47abb6a5ee35c3509a9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } }
I20260929 16:41:29.588512 3049 leader_election.cc:288] T 10e06387b28546a8b69e7c2d9168fee4 P 45f0267549da47abb6a5ee35c3509a9a [CANDIDATE]: Term 1 election: Requested vote from peers 84d0189b4c7342218cf6f788923580d4 (127.0.197.65:43273), c8d5e1e737d147d5b6d47fe948fc3623 (127.0.197.67:46267)
I20260929 16:41:29.589283 2972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e06387b28546a8b69e7c2d9168fee4" candidate_uuid: "45f0267549da47abb6a5ee35c3509a9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d5e1e737d147d5b6d47fe948fc3623"
I20260929 16:41:29.589275 2692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10e06387b28546a8b69e7c2d9168fee4" candidate_uuid: "45f0267549da47abb6a5ee35c3509a9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d0189b4c7342218cf6f788923580d4"
I20260929 16:41:29.589557 2692 raft_consensus.cc:3037] T 10e06387b28546a8b69e7c2d9168fee4 P 84d0189b4c7342218cf6f788923580d4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:41:29.589531 2972 raft_consensus.cc:3037] T 10e06387b28546a8b69e7c2d9168fee4 P c8d5e1e737d147d5b6d47fe948fc3623 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:41:29.593003 2692 raft_consensus.cc:2454] T 10e06387b28546a8b69e7c2d9168fee4 P 84d0189b4c7342218cf6f788923580d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45f0267549da47abb6a5ee35c3509a9a in term 1.
I20260929 16:41:29.593012 2972 raft_consensus.cc:2454] T 10e06387b28546a8b69e7c2d9168fee4 P c8d5e1e737d147d5b6d47fe948fc3623 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45f0267549da47abb6a5ee35c3509a9a in term 1.
I20260929 16:41:29.593559 2763 leader_election.cc:302] T 10e06387b28546a8b69e7c2d9168fee4 P 45f0267549da47abb6a5ee35c3509a9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45f0267549da47abb6a5ee35c3509a9a, 84d0189b4c7342218cf6f788923580d4; no voters:
I20260929 16:41:29.594007 3049 raft_consensus.cc:2781] T 10e06387b28546a8b69e7c2d9168fee4 P 45f0267549da47abb6a5ee35c3509a9a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:41:29.594558 3049 raft_consensus.cc:686] T 10e06387b28546a8b69e7c2d9168fee4 P 45f0267549da47abb6a5ee35c3509a9a [term 1 LEADER]: Becoming Leader. State: Replica: 45f0267549da47abb6a5ee35c3509a9a, State: Running, Role: LEADER
I20260929 16:41:29.595176 3049 consensus_queue.cc:237] T 10e06387b28546a8b69e7c2d9168fee4 P 45f0267549da47abb6a5ee35c3509a9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } }
I20260929 16:41:29.600217 2534 catalog_manager.cc:5495] T 10e06387b28546a8b69e7c2d9168fee4 P 45f0267549da47abb6a5ee35c3509a9a reported cstate change: term changed from 0 to 1, leader changed from <none> to 45f0267549da47abb6a5ee35c3509a9a (127.0.197.66). New cstate: current_term: 1 leader_uuid: "45f0267549da47abb6a5ee35c3509a9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } health_report { overall_health: HEALTHY } } }
Sep 29 16:41:29 dist-test-slave-l31f krb5kdc[2490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700087, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:41:29.732640 3060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:29.717968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44986 (local address 127.0.197.126:45257)
0929 16:41:29.719076 (+ 1108us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:29.719080 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:29.719101 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:41:29.719144 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:29.719147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:29.719201 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:29.719295 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:29.720569 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:29.721342 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:29.722670 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:29.723195 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:29.723338 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:29.727073 (+ 3735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:29.727081 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:29.727086 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:29.727126 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:29.729795 (+ 2669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:29.730578 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:29.730583 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:29.730586 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:29.730713 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:29.731125 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:29.731134 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:29.731137 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:29.731399 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:29.731557 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:29.731971 (+ 414us) server_negotiation.cc:299] Negotiation successful
0929 16:41:29.732212 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":946,"thread_start_us":829,"threads_started":1}
I20260929 16:41:29.732758 3059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:29.717872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:45257 (local address 127.0.0.1:44986)
0929 16:41:29.718570 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:29.718592 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:29.718724 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:29.719385 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:29.719391 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:29.720105 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:41:29.720349 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:29.720358 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:29.721490 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:29.721494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:29.722521 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:29.722533 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:29.723390 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:29.723397 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:29.723629 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:29.724424 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:41:29.724442 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:41:29.726764 (+ 2322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:41:29.729980 (+ 3216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:29.729995 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:29.730004 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:29.730439 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:29.730828 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:29.730832 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:29.730836 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:29.730981 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:29.731545 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:41:29.731554 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:41:29.731722 (+ 168us) client_negotiation.cc:769] Sending connection context
0929 16:41:29.731884 (+ 162us) client_negotiation.cc:240] Negotiation successful
0929 16:41:29.732340 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"mutex_wait_us":141,"thread_start_us":389,"threads_started":1}
I20260929 16:41:29.738792 2534 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:44986:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 16:41:29.740044 2534 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:41:29.747745 2952 tablet_service.cc:1467] Processing CreateTablet for tablet 1ba4b1e3e7534f6f8e00d0b4ab656181 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c3402b2d7dd5484ebe736a3096953ea8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 16:41:29.747575 2812 tablet_service.cc:1467] Processing CreateTablet for tablet 1ba4b1e3e7534f6f8e00d0b4ab656181 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c3402b2d7dd5484ebe736a3096953ea8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 16:41:29.747575 2672 tablet_service.cc:1467] Processing CreateTablet for tablet 1ba4b1e3e7534f6f8e00d0b4ab656181 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c3402b2d7dd5484ebe736a3096953ea8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 16:41:29.748577 2952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ba4b1e3e7534f6f8e00d0b4ab656181. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:41:29.748584 2672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ba4b1e3e7534f6f8e00d0b4ab656181. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:41:29.748584 2812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ba4b1e3e7534f6f8e00d0b4ab656181. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:41:29.755280 3042 tablet_bootstrap.cc:492] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P c8d5e1e737d147d5b6d47fe948fc3623: Bootstrap starting.
I20260929 16:41:29.756431 3041 tablet_bootstrap.cc:492] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P 84d0189b4c7342218cf6f788923580d4: Bootstrap starting.
I20260929 16:41:29.756589 3044 tablet_bootstrap.cc:492] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P 45f0267549da47abb6a5ee35c3509a9a: Bootstrap starting.
I20260929 16:41:29.757257 3042 tablet_bootstrap.cc:654] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P c8d5e1e737d147d5b6d47fe948fc3623: Neither blocks nor log segments found. Creating new log.
I20260929 16:41:29.758208 3044 tablet_bootstrap.cc:654] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P 45f0267549da47abb6a5ee35c3509a9a: Neither blocks nor log segments found. Creating new log.
I20260929 16:41:29.758208 3041 tablet_bootstrap.cc:654] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P 84d0189b4c7342218cf6f788923580d4: Neither blocks nor log segments found. Creating new log.
I20260929 16:41:29.759862 3042 tablet_bootstrap.cc:492] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P c8d5e1e737d147d5b6d47fe948fc3623: No bootstrap required, opened a new log
I20260929 16:41:29.760083 3042 ts_tablet_manager.cc:1397] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P c8d5e1e737d147d5b6d47fe948fc3623: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260929 16:41:29.760840 3041 tablet_bootstrap.cc:492] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P 84d0189b4c7342218cf6f788923580d4: No bootstrap required, opened a new log
I20260929 16:41:29.760996 3041 ts_tablet_manager.cc:1397] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P 84d0189b4c7342218cf6f788923580d4: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 16:41:29.761264 3044 tablet_bootstrap.cc:492] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P 45f0267549da47abb6a5ee35c3509a9a: No bootstrap required, opened a new log
I20260929 16:41:29.761112 3042 raft_consensus.cc:348] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P c8d5e1e737d147d5b6d47fe948fc3623 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } }
I20260929 16:41:29.761394 3042 raft_consensus.cc:374] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P c8d5e1e737d147d5b6d47fe948fc3623 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:41:29.761420 3044 ts_tablet_manager.cc:1397] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P 45f0267549da47abb6a5ee35c3509a9a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 16:41:29.761543 3042 raft_consensus.cc:729] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P c8d5e1e737d147d5b6d47fe948fc3623 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d5e1e737d147d5b6d47fe948fc3623, State: Initialized, Role: FOLLOWER
I20260929 16:41:29.761883 3042 consensus_queue.cc:260] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P c8d5e1e737d147d5b6d47fe948fc3623 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } }
I20260929 16:41:29.761884 3041 raft_consensus.cc:348] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P 84d0189b4c7342218cf6f788923580d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } }
I20260929 16:41:29.762140 3041 raft_consensus.cc:374] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P 84d0189b4c7342218cf6f788923580d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:41:29.762234 3041 raft_consensus.cc:729] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P 84d0189b4c7342218cf6f788923580d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84d0189b4c7342218cf6f788923580d4, State: Initialized, Role: FOLLOWER
I20260929 16:41:29.762401 3048 tablet_replica.cc:439] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P c8d5e1e737d147d5b6d47fe948fc3623: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } } }
I20260929 16:41:29.762496 3041 consensus_queue.cc:260] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P 84d0189b4c7342218cf6f788923580d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } }
I20260929 16:41:29.762472 3044 raft_consensus.cc:348] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P 45f0267549da47abb6a5ee35c3509a9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } }
I20260929 16:41:29.762728 3048 tablet_replica.cc:442] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P c8d5e1e737d147d5b6d47fe948fc3623: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:41:29.762741 3044 raft_consensus.cc:374] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P 45f0267549da47abb6a5ee35c3509a9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:41:29.762835 3044 raft_consensus.cc:729] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P 45f0267549da47abb6a5ee35c3509a9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45f0267549da47abb6a5ee35c3509a9a, State: Initialized, Role: FOLLOWER
I20260929 16:41:29.763099 3044 consensus_queue.cc:260] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P 45f0267549da47abb6a5ee35c3509a9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } }
I20260929 16:41:29.763134 3041 ts_tablet_manager.cc:1428] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P 84d0189b4c7342218cf6f788923580d4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 16:41:29.763557 3049 tablet_replica.cc:439] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P 45f0267549da47abb6a5ee35c3509a9a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } } }
I20260929 16:41:29.763901 3049 tablet_replica.cc:442] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P 45f0267549da47abb6a5ee35c3509a9a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:41:29.763582 3047 tablet_replica.cc:439] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P 84d0189b4c7342218cf6f788923580d4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } } }
I20260929 16:41:29.764002 3047 tablet_replica.cc:442] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P 84d0189b4c7342218cf6f788923580d4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:41:29.764765 3044 ts_tablet_manager.cc:1428] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P 45f0267549da47abb6a5ee35c3509a9a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 16:41:29.763305 3042 ts_tablet_manager.cc:1428] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P c8d5e1e737d147d5b6d47fe948fc3623: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 16:41:29.909914 3048 raft_consensus.cc:482] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P c8d5e1e737d147d5b6d47fe948fc3623 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:41:29.910181 3048 raft_consensus.cc:504] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P c8d5e1e737d147d5b6d47fe948fc3623 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } }
I20260929 16:41:29.912122 3048 leader_election.cc:288] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P c8d5e1e737d147d5b6d47fe948fc3623 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84d0189b4c7342218cf6f788923580d4 (127.0.197.65:43273), 45f0267549da47abb6a5ee35c3509a9a (127.0.197.66:43603)
I20260929 16:41:29.921206 3035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:29.912977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:49111 (local address 127.0.197.65:43273)
0929 16:41:29.913323 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:29.913330 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:29.913596 (+ 266us) server_negotiation.cc:407] Connection header received
0929 16:41:29.913942 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:29.913948 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:29.914045 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:29.914147 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:41:29.914830 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:29.915738 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:29.918152 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:29.919884 (+ 1732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:29.920135 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:29.920240 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:29.920699 (+ 459us) server_negotiation.cc:299] Negotiation successful
0929 16:41:29.920877 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183}
I20260929 16:41:29.921258 3066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:29.913979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:49939 (local address 127.0.197.66:43603)
0929 16:41:29.914649 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:29.914653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:29.914670 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:41:29.914757 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:29.914764 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:29.914842 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:29.914962 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:41:29.915793 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:29.916551 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:29.918825 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:29.920141 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:29.920323 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:29.920373 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:29.920812 (+ 439us) server_negotiation.cc:299] Negotiation successful
0929 16:41:29.920946 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":449,"threads_started":1}
I20260929 16:41:29.921322 3064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:29.912795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:43273 (local address 127.0.197.67:49111)
0929 16:41:29.913473 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:29.913489 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:29.913707 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:29.914342 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:29.914347 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:29.914372 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:41:29.914663 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:29.914673 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:29.915975 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:29.915979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:29.917847 (+ 1868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:29.917864 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:29.920144 (+ 2280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:29.920149 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:29.920480 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:29.920524 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 16:41:29.920647 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 16:41:29.920837 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"mutex_wait_us":74,"thread_start_us":437,"threads_started":1}
I20260929 16:41:29.921587 3065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:29.913821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.66:43603 (local address 127.0.197.67:49939)
0929 16:41:29.914395 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:29.914416 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:29.914508 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:29.915153 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:29.915158 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:29.915190 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:41:29.915598 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:29.915611 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:29.916708 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:29.916714 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:29.918481 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:29.918491 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:29.920405 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:29.920411 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:29.920647 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:29.920675 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 16:41:29.920755 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 16:41:29.920905 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"mutex_wait_us":61,"thread_start_us":254,"threads_started":1}
I20260929 16:41:29.922217 2692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ba4b1e3e7534f6f8e00d0b4ab656181" candidate_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d0189b4c7342218cf6f788923580d4" is_pre_election: true
I20260929 16:41:29.922523 2692 raft_consensus.cc:2454] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P 84d0189b4c7342218cf6f788923580d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d5e1e737d147d5b6d47fe948fc3623 in term 0.
I20260929 16:41:29.922621 2837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ba4b1e3e7534f6f8e00d0b4ab656181" candidate_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45f0267549da47abb6a5ee35c3509a9a" is_pre_election: true
I20260929 16:41:29.923182 2903 leader_election.cc:302] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P c8d5e1e737d147d5b6d47fe948fc3623 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84d0189b4c7342218cf6f788923580d4, c8d5e1e737d147d5b6d47fe948fc3623; no voters:
I20260929 16:41:29.923229 2837 raft_consensus.cc:2454] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P 45f0267549da47abb6a5ee35c3509a9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8d5e1e737d147d5b6d47fe948fc3623 in term 0.
I20260929 16:41:29.923658 3048 raft_consensus.cc:2781] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P c8d5e1e737d147d5b6d47fe948fc3623 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:41:29.923842 3048 raft_consensus.cc:482] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P c8d5e1e737d147d5b6d47fe948fc3623 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:41:29.923992 3048 raft_consensus.cc:3037] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P c8d5e1e737d147d5b6d47fe948fc3623 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:41:29.926220 3048 raft_consensus.cc:504] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P c8d5e1e737d147d5b6d47fe948fc3623 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } }
I20260929 16:41:29.926867 3048 leader_election.cc:288] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P c8d5e1e737d147d5b6d47fe948fc3623 [CANDIDATE]: Term 1 election: Requested vote from peers 84d0189b4c7342218cf6f788923580d4 (127.0.197.65:43273), 45f0267549da47abb6a5ee35c3509a9a (127.0.197.66:43603)
I20260929 16:41:29.927770 2692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ba4b1e3e7534f6f8e00d0b4ab656181" candidate_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d0189b4c7342218cf6f788923580d4"
I20260929 16:41:29.928103 2837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ba4b1e3e7534f6f8e00d0b4ab656181" candidate_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45f0267549da47abb6a5ee35c3509a9a"
I20260929 16:41:29.928239 2692 raft_consensus.cc:3037] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P 84d0189b4c7342218cf6f788923580d4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:41:29.928371 2837 raft_consensus.cc:3037] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P 45f0267549da47abb6a5ee35c3509a9a [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:41:29.930444 2692 raft_consensus.cc:2454] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P 84d0189b4c7342218cf6f788923580d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d5e1e737d147d5b6d47fe948fc3623 in term 1.
I20260929 16:41:29.931116 2903 leader_election.cc:302] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P c8d5e1e737d147d5b6d47fe948fc3623 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84d0189b4c7342218cf6f788923580d4, c8d5e1e737d147d5b6d47fe948fc3623; no voters:
I20260929 16:41:29.931702 3048 raft_consensus.cc:2781] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P c8d5e1e737d147d5b6d47fe948fc3623 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:41:29.931756 2837 raft_consensus.cc:2454] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P 45f0267549da47abb6a5ee35c3509a9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8d5e1e737d147d5b6d47fe948fc3623 in term 1.
I20260929 16:41:29.932286 3048 raft_consensus.cc:686] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P c8d5e1e737d147d5b6d47fe948fc3623 [term 1 LEADER]: Becoming Leader. State: Replica: c8d5e1e737d147d5b6d47fe948fc3623, State: Running, Role: LEADER
I20260929 16:41:29.932952 3048 consensus_queue.cc:237] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P c8d5e1e737d147d5b6d47fe948fc3623 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } }
I20260929 16:41:29.936600 3067 tablet_replica.cc:439] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P c8d5e1e737d147d5b6d47fe948fc3623: TxnStatusTablet state changed. Reason: New leader c8d5e1e737d147d5b6d47fe948fc3623. Latest consensus state: current_term: 1 leader_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } } }
I20260929 16:41:29.937072 3067 tablet_replica.cc:442] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P c8d5e1e737d147d5b6d47fe948fc3623: This TxnStatusTablet replica's current role is: LEADER
I20260929 16:41:29.937829 3069 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 16:41:29.938108 3069 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 16:41:29.938135 2534 catalog_manager.cc:5495] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P c8d5e1e737d147d5b6d47fe948fc3623 reported cstate change: term changed from 0 to 1, leader changed from <none> to c8d5e1e737d147d5b6d47fe948fc3623 (127.0.197.67). New cstate: current_term: 1 leader_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } health_report { overall_health: HEALTHY } } }
I20260929 16:41:29.966884 3037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:29.959657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40114 (local address 127.0.197.67:46267)
0929 16:41:29.959838 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:29.959843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:29.959915 (+ 72us) server_negotiation.cc:407] Connection header received
0929 16:41:29.960420 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:29.960426 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:29.960700 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:29.960854 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:41:29.961343 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:29.962341 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:29.964375 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:29.964867 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:29.964986 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:29.965629 (+ 643us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:41:29.966262 (+ 633us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:41:29.966402 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:29.966553 (+ 151us) server_negotiation.cc:299] Negotiation successful
0929 16:41:29.966629 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:41:29.967064 3059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:29.959544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.67:46267 (local address 127.0.0.1:40114)
0929 16:41:29.959804 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:29.959826 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:29.960021 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:29.960918 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:29.960924 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:29.960940 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:41:29.961181 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:29.961193 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:29.962487 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:29.962493 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:29.964187 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:29.964204 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:29.965071 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:29.965080 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:29.965304 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:29.965415 (+ 111us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:41:29.966399 (+ 984us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:41:29.966426 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 16:41:29.966523 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 16:41:29.966701 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 16:41:29.976447 2692 raft_consensus.cc:1261] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P 84d0189b4c7342218cf6f788923580d4 [term 1 FOLLOWER]: Refusing update from remote peer c8d5e1e737d147d5b6d47fe948fc3623: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:41:29.976475 2837 raft_consensus.cc:1261] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P 45f0267549da47abb6a5ee35c3509a9a [term 1 FOLLOWER]: Refusing update from remote peer c8d5e1e737d147d5b6d47fe948fc3623: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:41:29.977520 3067 consensus_queue.cc:1035] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P c8d5e1e737d147d5b6d47fe948fc3623 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:41:29.977890 3048 consensus_queue.cc:1035] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P c8d5e1e737d147d5b6d47fe948fc3623 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:41:29.995616 3047 tablet_replica.cc:439] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P 84d0189b4c7342218cf6f788923580d4: TxnStatusTablet state changed. Reason: New leader c8d5e1e737d147d5b6d47fe948fc3623. Latest consensus state: current_term: 1 leader_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } } }
I20260929 16:41:29.995909 3047 tablet_replica.cc:442] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P 84d0189b4c7342218cf6f788923580d4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:41:29.997179 3049 tablet_replica.cc:439] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P 45f0267549da47abb6a5ee35c3509a9a: TxnStatusTablet state changed. Reason: New leader c8d5e1e737d147d5b6d47fe948fc3623. Latest consensus state: current_term: 1 leader_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } } }
I20260929 16:41:29.997530 3049 tablet_replica.cc:442] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P 45f0267549da47abb6a5ee35c3509a9a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:41:29.999435 3072 mvcc.cc:205] Tried to move back new op lower bound from 7334707568531787776 to 7334707568369328128. Current Snapshot: MvccSnapshot[applied={T|T < 7334707568531787776}]
I20260929 16:41:30.005651 3067 tablet_replica.cc:439] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P c8d5e1e737d147d5b6d47fe948fc3623: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } } }
I20260929 16:41:30.006258 3067 tablet_replica.cc:442] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P c8d5e1e737d147d5b6d47fe948fc3623: This TxnStatusTablet replica's current role is: LEADER
I20260929 16:41:30.009184 3067 tablet_replica.cc:439] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P c8d5e1e737d147d5b6d47fe948fc3623: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } } }
I20260929 16:41:30.009694 3067 tablet_replica.cc:442] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P c8d5e1e737d147d5b6d47fe948fc3623: This TxnStatusTablet replica's current role is: LEADER
I20260929 16:41:30.034861 3049 tablet_replica.cc:439] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P 45f0267549da47abb6a5ee35c3509a9a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } } }
I20260929 16:41:30.035341 3049 tablet_replica.cc:442] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P 45f0267549da47abb6a5ee35c3509a9a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:41:30.040536 3047 tablet_replica.cc:439] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P 84d0189b4c7342218cf6f788923580d4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } } }
I20260929 16:41:30.040848 3047 tablet_replica.cc:442] T 1ba4b1e3e7534f6f8e00d0b4ab656181 P 84d0189b4c7342218cf6f788923580d4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:41:30.041712 3066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:30.032882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43724 (local address 127.0.197.66:43603)
0929 16:41:30.033136 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:30.033143 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:30.034055 (+ 912us) server_negotiation.cc:407] Connection header received
0929 16:41:30.034426 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:30.034431 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:30.034543 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:30.034690 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:41:30.035431 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:30.036463 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:30.038604 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:30.039319 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:30.039431 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:30.040113 (+ 682us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:41:30.040886 (+ 773us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:41:30.041013 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:30.041272 (+ 259us) server_negotiation.cc:299] Negotiation successful
0929 16:41:30.041396 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 16:41:30.060386 3037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:30.054082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:34647 (local address 127.0.197.67:46267)
0929 16:41:30.054325 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:30.054334 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:30.054790 (+ 456us) server_negotiation.cc:407] Connection header received
0929 16:41:30.055075 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:30.055080 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:30.055152 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:30.055260 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:41:30.055756 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:30.056818 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:30.058409 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:30.058896 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:30.059075 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:30.059569 (+ 494us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:41:30.059693 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:41:30.059814 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:30.059962 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 16:41:30.060051 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 16:41:30.060500 3085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:30.053989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.67:46267 (local address 127.0.197.66:34647)
0929 16:41:30.054675 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:30.054692 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:30.054876 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:30.055291 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:30.055295 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:30.055308 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:41:30.055600 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:30.055609 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:30.056999 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:30.057005 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:30.058248 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:30.058263 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:30.059111 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:30.059119 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:30.059259 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:30.059327 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:41:30.059815 (+ 488us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:41:30.059836 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 16:41:30.059938 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 16:41:30.060089 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":387,"threads_started":1}
I20260929 16:41:30.071489 2972 raft_consensus.cc:1261] T 10e06387b28546a8b69e7c2d9168fee4 P c8d5e1e737d147d5b6d47fe948fc3623 [term 1 FOLLOWER]: Refusing update from remote peer 45f0267549da47abb6a5ee35c3509a9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:41:30.071537 2692 raft_consensus.cc:1261] T 10e06387b28546a8b69e7c2d9168fee4 P 84d0189b4c7342218cf6f788923580d4 [term 1 FOLLOWER]: Refusing update from remote peer 45f0267549da47abb6a5ee35c3509a9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:41:30.072166 3052 consensus_queue.cc:1035] T 10e06387b28546a8b69e7c2d9168fee4 P 45f0267549da47abb6a5ee35c3509a9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:41:30.072907 3049 consensus_queue.cc:1035] T 10e06387b28546a8b69e7c2d9168fee4 P 45f0267549da47abb6a5ee35c3509a9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:41:30.079777 3073 mvcc.cc:205] Tried to move back new op lower bound from 7334707568925429760 to 7334707566985981952. Current Snapshot: MvccSnapshot[applied={T|T < 7334707568925429760}]
I20260929 16:41:30.110484 3066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:30.102870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:50827 (local address 127.0.197.66:43603)
0929 16:41:30.103104 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:30.103111 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:30.103582 (+ 471us) server_negotiation.cc:407] Connection header received
0929 16:41:30.103810 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:30.103816 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:30.103909 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:30.104019 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:41:30.104821 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:30.105819 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:30.108172 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:30.108881 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:30.109042 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:30.109687 (+ 645us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:41:30.109826 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:41:30.109925 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:30.110084 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 16:41:30.110171 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 16:41:30.111526 3089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:30.102752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.66:43603 (local address 127.0.197.67:50827)
0929 16:41:30.103464 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:30.103481 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:30.103647 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:30.104186 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:30.104194 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:30.104208 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:41:30.104601 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:30.104613 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:30.105966 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:30.105972 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:30.107927 (+ 1955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:30.107943 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:30.109154 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:30.109163 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:30.109364 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:30.109475 (+ 111us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:41:30.109940 (+ 465us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:41:30.109952 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 16:41:30.110209 (+ 257us) client_negotiation.cc:240] Negotiation successful
0929 16:41:30.110385 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":358,"threads_started":1}
I20260929 16:41:30.160853 2534 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:41:30.161195 2534 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:41:30.165593 2534 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e8c03d81d4e8437d90edd6d0e53056a5: Sending DeleteTablet for 3 replicas of tablet 10e06387b28546a8b69e7c2d9168fee4
I20260929 16:41:30.166860 2672 tablet_service.cc:1514] Processing DeleteTablet for tablet 10e06387b28546a8b69e7c2d9168fee4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:41:30 UTC) from {username='kudu', principal='kudu/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:59056
I20260929 16:41:30.167760 2952 tablet_service.cc:1514] Processing DeleteTablet for tablet 10e06387b28546a8b69e7c2d9168fee4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:41:30 UTC) from {username='kudu', principal='kudu/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:40104
I20260929 16:41:30.168778 2812 tablet_service.cc:1514] Processing DeleteTablet for tablet 10e06387b28546a8b69e7c2d9168fee4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:41:30 UTC) from {username='kudu', principal='kudu/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:43720
I20260929 16:41:30.169623 2534 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44982:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:41:30.170711 2534 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:41:30.181764 2951 tablet_service.cc:1467] Processing CreateTablet for tablet 42d11bb37b964efd8f08e811def08318 (DEFAULT_TABLE table=test-table [id=10c0487a1a514276b01d8894db3bc48d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:41:30.182058 2671 tablet_service.cc:1467] Processing CreateTablet for tablet 42d11bb37b964efd8f08e811def08318 (DEFAULT_TABLE table=test-table [id=10c0487a1a514276b01d8894db3bc48d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:41:30.182492 2951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42d11bb37b964efd8f08e811def08318. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:41:30.182636 2671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42d11bb37b964efd8f08e811def08318. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:41:30.183985 3094 tablet_replica.cc:330] T 10e06387b28546a8b69e7c2d9168fee4 P c8d5e1e737d147d5b6d47fe948fc3623: stopping tablet replica
I20260929 16:41:30.184574 3094 raft_consensus.cc:2229] T 10e06387b28546a8b69e7c2d9168fee4 P c8d5e1e737d147d5b6d47fe948fc3623 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:41:30.188897 2811 tablet_service.cc:1467] Processing CreateTablet for tablet 42d11bb37b964efd8f08e811def08318 (DEFAULT_TABLE table=test-table [id=10c0487a1a514276b01d8894db3bc48d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:41:30.189568 2811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42d11bb37b964efd8f08e811def08318. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:41:30.190552 3042 tablet_bootstrap.cc:492] T 42d11bb37b964efd8f08e811def08318 P c8d5e1e737d147d5b6d47fe948fc3623: Bootstrap starting.
I20260929 16:41:30.191404 3094 raft_consensus.cc:2258] T 10e06387b28546a8b69e7c2d9168fee4 P c8d5e1e737d147d5b6d47fe948fc3623 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:41:30.192572 3042 tablet_bootstrap.cc:654] T 42d11bb37b964efd8f08e811def08318 P c8d5e1e737d147d5b6d47fe948fc3623: Neither blocks nor log segments found. Creating new log.
I20260929 16:41:30.195194 3042 tablet_bootstrap.cc:492] T 42d11bb37b964efd8f08e811def08318 P c8d5e1e737d147d5b6d47fe948fc3623: No bootstrap required, opened a new log
I20260929 16:41:30.195495 3042 ts_tablet_manager.cc:1397] T 42d11bb37b964efd8f08e811def08318 P c8d5e1e737d147d5b6d47fe948fc3623: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 16:41:30.196705 3042 raft_consensus.cc:348] T 42d11bb37b964efd8f08e811def08318 P c8d5e1e737d147d5b6d47fe948fc3623 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } }
I20260929 16:41:30.197120 3042 raft_consensus.cc:374] T 42d11bb37b964efd8f08e811def08318 P c8d5e1e737d147d5b6d47fe948fc3623 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:41:30.197285 3042 raft_consensus.cc:729] T 42d11bb37b964efd8f08e811def08318 P c8d5e1e737d147d5b6d47fe948fc3623 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d5e1e737d147d5b6d47fe948fc3623, State: Initialized, Role: FOLLOWER
I20260929 16:41:30.197687 3042 consensus_queue.cc:260] T 42d11bb37b964efd8f08e811def08318 P c8d5e1e737d147d5b6d47fe948fc3623 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } }
I20260929 16:41:30.198957 3042 ts_tablet_manager.cc:1428] T 42d11bb37b964efd8f08e811def08318 P c8d5e1e737d147d5b6d47fe948fc3623: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 16:41:30.199436 3094 ts_tablet_manager.cc:1905] T 10e06387b28546a8b69e7c2d9168fee4 P c8d5e1e737d147d5b6d47fe948fc3623: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:41:30.203562 3044 tablet_bootstrap.cc:492] T 42d11bb37b964efd8f08e811def08318 P 45f0267549da47abb6a5ee35c3509a9a: Bootstrap starting.
I20260929 16:41:30.205384 3044 tablet_bootstrap.cc:654] T 42d11bb37b964efd8f08e811def08318 P 45f0267549da47abb6a5ee35c3509a9a: Neither blocks nor log segments found. Creating new log.
I20260929 16:41:30.207896 3044 tablet_bootstrap.cc:492] T 42d11bb37b964efd8f08e811def08318 P 45f0267549da47abb6a5ee35c3509a9a: No bootstrap required, opened a new log
I20260929 16:41:30.208115 3044 ts_tablet_manager.cc:1397] T 42d11bb37b964efd8f08e811def08318 P 45f0267549da47abb6a5ee35c3509a9a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 16:41:30.208863 3044 raft_consensus.cc:348] T 42d11bb37b964efd8f08e811def08318 P 45f0267549da47abb6a5ee35c3509a9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } }
I20260929 16:41:30.209146 3044 raft_consensus.cc:374] T 42d11bb37b964efd8f08e811def08318 P 45f0267549da47abb6a5ee35c3509a9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:41:30.209233 3044 raft_consensus.cc:729] T 42d11bb37b964efd8f08e811def08318 P 45f0267549da47abb6a5ee35c3509a9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45f0267549da47abb6a5ee35c3509a9a, State: Initialized, Role: FOLLOWER
I20260929 16:41:30.209555 3044 consensus_queue.cc:260] T 42d11bb37b964efd8f08e811def08318 P 45f0267549da47abb6a5ee35c3509a9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } }
I20260929 16:41:30.210947 3044 ts_tablet_manager.cc:1428] T 42d11bb37b964efd8f08e811def08318 P 45f0267549da47abb6a5ee35c3509a9a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 16:41:30.217164 3041 tablet_bootstrap.cc:492] T 42d11bb37b964efd8f08e811def08318 P 84d0189b4c7342218cf6f788923580d4: Bootstrap starting.
I20260929 16:41:30.218077 3094 ts_tablet_manager.cc:1918] T 10e06387b28546a8b69e7c2d9168fee4 P c8d5e1e737d147d5b6d47fe948fc3623: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:41:30.218295 3094 log.cc:1196] T 10e06387b28546a8b69e7c2d9168fee4 P c8d5e1e737d147d5b6d47fe948fc3623: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-2/wal/wals/10e06387b28546a8b69e7c2d9168fee4
I20260929 16:41:30.218948 3094 ts_tablet_manager.cc:1939] T 10e06387b28546a8b69e7c2d9168fee4 P c8d5e1e737d147d5b6d47fe948fc3623: Deleting consensus metadata
I20260929 16:41:30.219488 3041 tablet_bootstrap.cc:654] T 42d11bb37b964efd8f08e811def08318 P 84d0189b4c7342218cf6f788923580d4: Neither blocks nor log segments found. Creating new log.
I20260929 16:41:30.221020 2522 catalog_manager.cc:4841] TS c8d5e1e737d147d5b6d47fe948fc3623 (127.0.197.67:46267): tablet 10e06387b28546a8b69e7c2d9168fee4 (table test-table [id=887b75c324d140b0b1aae84b76632a10]) successfully deleted
I20260929 16:41:30.229765 3095 tablet_replica.cc:330] T 10e06387b28546a8b69e7c2d9168fee4 P 45f0267549da47abb6a5ee35c3509a9a: stopping tablet replica
I20260929 16:41:30.230319 3095 raft_consensus.cc:2229] T 10e06387b28546a8b69e7c2d9168fee4 P 45f0267549da47abb6a5ee35c3509a9a [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:41:30.231205 3095 raft_consensus.cc:2258] T 10e06387b28546a8b69e7c2d9168fee4 P 45f0267549da47abb6a5ee35c3509a9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:41:30.234058 3093 tablet_replica.cc:330] T 10e06387b28546a8b69e7c2d9168fee4 P 84d0189b4c7342218cf6f788923580d4: stopping tablet replica
I20260929 16:41:30.234681 3093 raft_consensus.cc:2229] T 10e06387b28546a8b69e7c2d9168fee4 P 84d0189b4c7342218cf6f788923580d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:41:30.235249 3093 raft_consensus.cc:2258] T 10e06387b28546a8b69e7c2d9168fee4 P 84d0189b4c7342218cf6f788923580d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:41:30.239326 3095 ts_tablet_manager.cc:1905] T 10e06387b28546a8b69e7c2d9168fee4 P 45f0267549da47abb6a5ee35c3509a9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:41:30.247458 3093 ts_tablet_manager.cc:1905] T 10e06387b28546a8b69e7c2d9168fee4 P 84d0189b4c7342218cf6f788923580d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:41:30.252575 3041 tablet_bootstrap.cc:492] T 42d11bb37b964efd8f08e811def08318 P 84d0189b4c7342218cf6f788923580d4: No bootstrap required, opened a new log
I20260929 16:41:30.252843 3041 ts_tablet_manager.cc:1397] T 42d11bb37b964efd8f08e811def08318 P 84d0189b4c7342218cf6f788923580d4: Time spent bootstrapping tablet: real 0.036s user 0.018s sys 0.009s
I20260929 16:41:30.253688 3041 raft_consensus.cc:348] T 42d11bb37b964efd8f08e811def08318 P 84d0189b4c7342218cf6f788923580d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } }
I20260929 16:41:30.254031 3041 raft_consensus.cc:374] T 42d11bb37b964efd8f08e811def08318 P 84d0189b4c7342218cf6f788923580d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:41:30.254144 3041 raft_consensus.cc:729] T 42d11bb37b964efd8f08e811def08318 P 84d0189b4c7342218cf6f788923580d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84d0189b4c7342218cf6f788923580d4, State: Initialized, Role: FOLLOWER
I20260929 16:41:30.254487 3041 consensus_queue.cc:260] T 42d11bb37b964efd8f08e811def08318 P 84d0189b4c7342218cf6f788923580d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } }
I20260929 16:41:30.257906 3041 ts_tablet_manager.cc:1428] T 42d11bb37b964efd8f08e811def08318 P 84d0189b4c7342218cf6f788923580d4: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20260929 16:41:30.265614 3047 raft_consensus.cc:482] T 42d11bb37b964efd8f08e811def08318 P 84d0189b4c7342218cf6f788923580d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:41:30.265829 3047 raft_consensus.cc:504] T 42d11bb37b964efd8f08e811def08318 P 84d0189b4c7342218cf6f788923580d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } }
I20260929 16:41:30.267598 3047 leader_election.cc:288] T 42d11bb37b964efd8f08e811def08318 P 84d0189b4c7342218cf6f788923580d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45f0267549da47abb6a5ee35c3509a9a (127.0.197.66:43603), c8d5e1e737d147d5b6d47fe948fc3623 (127.0.197.67:46267)
I20260929 16:41:30.275120 3093 ts_tablet_manager.cc:1918] T 10e06387b28546a8b69e7c2d9168fee4 P 84d0189b4c7342218cf6f788923580d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:41:30.275352 3093 log.cc:1196] T 10e06387b28546a8b69e7c2d9168fee4 P 84d0189b4c7342218cf6f788923580d4: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-0/wal/wals/10e06387b28546a8b69e7c2d9168fee4
I20260929 16:41:30.276064 3093 ts_tablet_manager.cc:1939] T 10e06387b28546a8b69e7c2d9168fee4 P 84d0189b4c7342218cf6f788923580d4: Deleting consensus metadata
I20260929 16:41:30.278599 2520 catalog_manager.cc:4841] TS 84d0189b4c7342218cf6f788923580d4 (127.0.197.65:43273): tablet 10e06387b28546a8b69e7c2d9168fee4 (table test-table [id=887b75c324d140b0b1aae84b76632a10]) successfully deleted
I20260929 16:41:30.278779 3037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:30.268771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:33813 (local address 127.0.197.67:46267)
0929 16:41:30.268995 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:30.269002 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:30.270435 (+ 1433us) server_negotiation.cc:407] Connection header received
0929 16:41:30.270761 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:30.270768 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:30.270846 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:30.270909 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:41:30.271785 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:30.272941 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:30.275898 (+ 2957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:30.277328 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:30.277499 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:30.277601 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:30.278299 (+ 698us) server_negotiation.cc:299] Negotiation successful
0929 16:41:30.278463 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 16:41:30.279184 3095 ts_tablet_manager.cc:1918] T 10e06387b28546a8b69e7c2d9168fee4 P 45f0267549da47abb6a5ee35c3509a9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:41:30.279384 3095 log.cc:1196] T 10e06387b28546a8b69e7c2d9168fee4 P 45f0267549da47abb6a5ee35c3509a9a: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-1/wal/wals/10e06387b28546a8b69e7c2d9168fee4
I20260929 16:41:30.280040 3095 ts_tablet_manager.cc:1939] T 10e06387b28546a8b69e7c2d9168fee4 P 45f0267549da47abb6a5ee35c3509a9a: Deleting consensus metadata
I20260929 16:41:30.282526 2520 catalog_manager.cc:4841] TS 45f0267549da47abb6a5ee35c3509a9a (127.0.197.66:43603): tablet 10e06387b28546a8b69e7c2d9168fee4 (table test-table [id=887b75c324d140b0b1aae84b76632a10]) successfully deleted
I20260929 16:41:30.283677 3099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:30.268268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.67:46267 (local address 127.0.197.65:33813)
0929 16:41:30.270312 (+ 2044us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:30.270334 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:30.270576 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:30.271129 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:30.271134 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:30.271170 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:41:30.271593 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:30.271606 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:30.273162 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:30.273167 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:30.275667 (+ 2500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:30.275687 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:30.277840 (+ 2153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:30.277848 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:30.278081 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:30.278118 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 16:41:30.283068 (+ 4950us) client_negotiation.cc:240] Negotiation successful
0929 16:41:30.283272 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1905,"thread_start_us":1778,"threads_started":1}
I20260929 16:41:30.284808 2972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42d11bb37b964efd8f08e811def08318" candidate_uuid: "84d0189b4c7342218cf6f788923580d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" is_pre_election: true
I20260929 16:41:30.285115 2972 raft_consensus.cc:2454] T 42d11bb37b964efd8f08e811def08318 P c8d5e1e737d147d5b6d47fe948fc3623 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84d0189b4c7342218cf6f788923580d4 in term 0.
I20260929 16:41:30.286418 2625 leader_election.cc:302] T 42d11bb37b964efd8f08e811def08318 P 84d0189b4c7342218cf6f788923580d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84d0189b4c7342218cf6f788923580d4, c8d5e1e737d147d5b6d47fe948fc3623; no voters:
I20260929 16:41:30.286942 3047 raft_consensus.cc:2781] T 42d11bb37b964efd8f08e811def08318 P 84d0189b4c7342218cf6f788923580d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:41:30.287124 3047 raft_consensus.cc:482] T 42d11bb37b964efd8f08e811def08318 P 84d0189b4c7342218cf6f788923580d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:41:30.287221 3047 raft_consensus.cc:3037] T 42d11bb37b964efd8f08e811def08318 P 84d0189b4c7342218cf6f788923580d4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:41:30.290807 3047 raft_consensus.cc:504] T 42d11bb37b964efd8f08e811def08318 P 84d0189b4c7342218cf6f788923580d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } }
I20260929 16:41:30.292434 3066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:30.269718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:54823 (local address 127.0.197.66:43603)
0929 16:41:30.269888 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:30.269899 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:30.283247 (+ 13348us) server_negotiation.cc:407] Connection header received
0929 16:41:30.283558 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:30.283563 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:30.283640 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:30.283717 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:41:30.285110 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:30.286004 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:30.289206 (+ 3202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:30.290684 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:30.290874 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:30.291147 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:30.291952 (+ 805us) server_negotiation.cc:299] Negotiation successful
0929 16:41:30.292154 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 16:41:30.292661 2972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42d11bb37b964efd8f08e811def08318" candidate_uuid: "84d0189b4c7342218cf6f788923580d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d5e1e737d147d5b6d47fe948fc3623"
I20260929 16:41:30.292894 2972 raft_consensus.cc:3037] T 42d11bb37b964efd8f08e811def08318 P c8d5e1e737d147d5b6d47fe948fc3623 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:41:30.293483 3100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:30.269372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.66:43603 (local address 127.0.197.65:54823)
0929 16:41:30.283120 (+ 13748us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:30.283141 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:30.283379 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:30.283944 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:30.283951 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:30.283981 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:41:30.284884 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:30.284898 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:30.286332 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:30.286337 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:30.288925 (+ 2588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:30.288947 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:30.291574 (+ 2627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:30.291582 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:30.291755 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:30.291788 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 16:41:30.292916 (+ 1128us) client_negotiation.cc:240] Negotiation successful
0929 16:41:30.293128 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13503,"thread_start_us":13445,"threads_started":1}
I20260929 16:41:30.293859 3047 leader_election.cc:288] T 42d11bb37b964efd8f08e811def08318 P 84d0189b4c7342218cf6f788923580d4 [CANDIDATE]: Term 1 election: Requested vote from peers 45f0267549da47abb6a5ee35c3509a9a (127.0.197.66:43603), c8d5e1e737d147d5b6d47fe948fc3623 (127.0.197.67:46267)
I20260929 16:41:30.294764 2837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42d11bb37b964efd8f08e811def08318" candidate_uuid: "84d0189b4c7342218cf6f788923580d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45f0267549da47abb6a5ee35c3509a9a" is_pre_election: true
I20260929 16:41:30.295092 2837 raft_consensus.cc:2454] T 42d11bb37b964efd8f08e811def08318 P 45f0267549da47abb6a5ee35c3509a9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84d0189b4c7342218cf6f788923580d4 in term 0.
I20260929 16:41:30.295038 2836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42d11bb37b964efd8f08e811def08318" candidate_uuid: "84d0189b4c7342218cf6f788923580d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45f0267549da47abb6a5ee35c3509a9a"
I20260929 16:41:30.295342 2836 raft_consensus.cc:3037] T 42d11bb37b964efd8f08e811def08318 P 45f0267549da47abb6a5ee35c3509a9a [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:41:30.298278 2836 raft_consensus.cc:2454] T 42d11bb37b964efd8f08e811def08318 P 45f0267549da47abb6a5ee35c3509a9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84d0189b4c7342218cf6f788923580d4 in term 1.
I20260929 16:41:30.298717 2972 raft_consensus.cc:2454] T 42d11bb37b964efd8f08e811def08318 P c8d5e1e737d147d5b6d47fe948fc3623 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84d0189b4c7342218cf6f788923580d4 in term 1.
I20260929 16:41:30.298883 2623 leader_election.cc:302] T 42d11bb37b964efd8f08e811def08318 P 84d0189b4c7342218cf6f788923580d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45f0267549da47abb6a5ee35c3509a9a, 84d0189b4c7342218cf6f788923580d4; no voters:
I20260929 16:41:30.300397 3047 raft_consensus.cc:2781] T 42d11bb37b964efd8f08e811def08318 P 84d0189b4c7342218cf6f788923580d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:41:30.300987 3047 raft_consensus.cc:686] T 42d11bb37b964efd8f08e811def08318 P 84d0189b4c7342218cf6f788923580d4 [term 1 LEADER]: Becoming Leader. State: Replica: 84d0189b4c7342218cf6f788923580d4, State: Running, Role: LEADER
I20260929 16:41:30.301606 3047 consensus_queue.cc:237] T 42d11bb37b964efd8f08e811def08318 P 84d0189b4c7342218cf6f788923580d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } }
I20260929 16:41:30.305550 2533 catalog_manager.cc:5495] T 42d11bb37b964efd8f08e811def08318 P 84d0189b4c7342218cf6f788923580d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84d0189b4c7342218cf6f788923580d4 (127.0.197.65). New cstate: current_term: 1 leader_uuid: "84d0189b4c7342218cf6f788923580d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } health_report { overall_health: UNKNOWN } } }
I20260929 16:41:30.347054 3035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:30.341254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59066 (local address 127.0.197.65:43273)
0929 16:41:30.341432 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:30.341438 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:30.341456 (+ 18us) server_negotiation.cc:407] Connection header received
0929 16:41:30.341563 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:30.341568 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:30.341678 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:30.341844 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:41:30.342418 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:30.343397 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:30.344754 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:30.345171 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:30.345304 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:30.345733 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:41:30.346331 (+ 598us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:41:30.346494 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:30.346604 (+ 110us) server_negotiation.cc:299] Negotiation successful
0929 16:41:30.346687 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 16:41:30.364363 3037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:30.358341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:42667 (local address 127.0.197.67:46267)
0929 16:41:30.358535 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:30.358542 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:30.358789 (+ 247us) server_negotiation.cc:407] Connection header received
0929 16:41:30.359091 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:30.359095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:30.359159 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:30.359262 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:41:30.359743 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:30.360480 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:30.362442 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:30.362904 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:30.363113 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:30.363576 (+ 463us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:41:30.363690 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:41:30.363794 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:30.363920 (+ 126us) server_negotiation.cc:299] Negotiation successful
0929 16:41:30.364000 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 16:41:30.364434 3105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:30.357970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.67:46267 (local address 127.0.197.65:42667)
0929 16:41:30.358674 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:30.358694 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:30.358902 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:30.359278 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:30.359283 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:30.359294 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:41:30.359599 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:30.359607 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:30.360635 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:30.360640 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:30.362239 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:30.362260 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:30.363136 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:30.363143 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:30.363295 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:30.363357 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:41:30.363792 (+ 435us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:41:30.363820 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 16:41:30.363900 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 16:41:30.364052 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":584,"threads_started":1}
I20260929 16:41:30.377954 2836 raft_consensus.cc:1261] T 42d11bb37b964efd8f08e811def08318 P 45f0267549da47abb6a5ee35c3509a9a [term 1 FOLLOWER]: Refusing update from remote peer 84d0189b4c7342218cf6f788923580d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:41:30.377918 2972 raft_consensus.cc:1261] T 42d11bb37b964efd8f08e811def08318 P c8d5e1e737d147d5b6d47fe948fc3623 [term 1 FOLLOWER]: Refusing update from remote peer 84d0189b4c7342218cf6f788923580d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:41:30.378750 3101 consensus_queue.cc:1035] T 42d11bb37b964efd8f08e811def08318 P 84d0189b4c7342218cf6f788923580d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:41:30.379604 3047 consensus_queue.cc:1035] T 42d11bb37b964efd8f08e811def08318 P 84d0189b4c7342218cf6f788923580d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:41:30.432463 3089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:30.425950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:43273 (local address 127.0.197.67:45473)
0929 16:41:30.426177 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:30.426201 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:30.426337 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:30.426924 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:30.426929 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:30.426942 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:41:30.427189 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:30.427200 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:30.428509 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:30.428514 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:30.430085 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:30.430102 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:30.431241 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:30.431247 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:30.431437 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:30.431507 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:41:30.431970 (+ 463us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:41:30.431984 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 16:41:30.432065 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 16:41:30.432195 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 16:41:30.433101 3035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:30.426274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:45473 (local address 127.0.197.65:43273)
0929 16:41:30.426454 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:30.426461 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:30.426481 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:41:30.426557 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:30.426562 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:30.426648 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:30.426746 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:41:30.427357 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:30.428341 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:30.430264 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:30.430962 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:30.431121 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:30.431698 (+ 577us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:41:30.431846 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:41:30.431965 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:30.432717 (+ 752us) server_negotiation.cc:299] Negotiation successful
0929 16:41:30.432799 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260929 16:41:30.434520 3073 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 16:41:32.057503 2534 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:41:32.057746 2534 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:41:32.063227 2534 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e8c03d81d4e8437d90edd6d0e53056a5: Sending DeleteTablet for 3 replicas of tablet 42d11bb37b964efd8f08e811def08318
I20260929 16:41:32.064280 2672 tablet_service.cc:1514] Processing DeleteTablet for tablet 42d11bb37b964efd8f08e811def08318 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:41:32 UTC) from {username='kudu', principal='kudu/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:59056
I20260929 16:41:32.064280 2952 tablet_service.cc:1514] Processing DeleteTablet for tablet 42d11bb37b964efd8f08e811def08318 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:41:32 UTC) from {username='kudu', principal='kudu/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:40104
I20260929 16:41:32.064792 2812 tablet_service.cc:1514] Processing DeleteTablet for tablet 42d11bb37b964efd8f08e811def08318 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:41:32 UTC) from {username='kudu', principal='kudu/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:43720
I20260929 16:41:32.077056 3131 tablet_replica.cc:330] T 42d11bb37b964efd8f08e811def08318 P 84d0189b4c7342218cf6f788923580d4: stopping tablet replica
I20260929 16:41:32.077364 3131 raft_consensus.cc:2229] T 42d11bb37b964efd8f08e811def08318 P 84d0189b4c7342218cf6f788923580d4 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:41:32.077934 3133 tablet_replica.cc:330] T 42d11bb37b964efd8f08e811def08318 P 45f0267549da47abb6a5ee35c3509a9a: stopping tablet replica
I20260929 16:41:32.078249 3133 raft_consensus.cc:2229] T 42d11bb37b964efd8f08e811def08318 P 45f0267549da47abb6a5ee35c3509a9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:41:32.078363 3131 raft_consensus.cc:2258] T 42d11bb37b964efd8f08e811def08318 P 84d0189b4c7342218cf6f788923580d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:41:32.078636 3133 raft_consensus.cc:2258] T 42d11bb37b964efd8f08e811def08318 P 45f0267549da47abb6a5ee35c3509a9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:41:32.079614 3131 ts_tablet_manager.cc:1905] T 42d11bb37b964efd8f08e811def08318 P 84d0189b4c7342218cf6f788923580d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:41:32.079698 3133 ts_tablet_manager.cc:1905] T 42d11bb37b964efd8f08e811def08318 P 45f0267549da47abb6a5ee35c3509a9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:41:32.080343 3132 tablet_replica.cc:330] T 42d11bb37b964efd8f08e811def08318 P c8d5e1e737d147d5b6d47fe948fc3623: stopping tablet replica
I20260929 16:41:32.080582 3132 raft_consensus.cc:2229] T 42d11bb37b964efd8f08e811def08318 P c8d5e1e737d147d5b6d47fe948fc3623 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:41:32.080910 3132 raft_consensus.cc:2258] T 42d11bb37b964efd8f08e811def08318 P c8d5e1e737d147d5b6d47fe948fc3623 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:41:32.082067 3132 ts_tablet_manager.cc:1905] T 42d11bb37b964efd8f08e811def08318 P c8d5e1e737d147d5b6d47fe948fc3623: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:41:32.087383 3133 ts_tablet_manager.cc:1918] T 42d11bb37b964efd8f08e811def08318 P 45f0267549da47abb6a5ee35c3509a9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 16:41:32.087580 3131 ts_tablet_manager.cc:1918] T 42d11bb37b964efd8f08e811def08318 P 84d0189b4c7342218cf6f788923580d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 16:41:32.087692 3133 log.cc:1196] T 42d11bb37b964efd8f08e811def08318 P 45f0267549da47abb6a5ee35c3509a9a: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-1/wal/wals/42d11bb37b964efd8f08e811def08318
I20260929 16:41:32.087765 3131 log.cc:1196] T 42d11bb37b964efd8f08e811def08318 P 84d0189b4c7342218cf6f788923580d4: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-0/wal/wals/42d11bb37b964efd8f08e811def08318
I20260929 16:41:32.088272 3133 ts_tablet_manager.cc:1939] T 42d11bb37b964efd8f08e811def08318 P 45f0267549da47abb6a5ee35c3509a9a: Deleting consensus metadata
I20260929 16:41:32.088279 3131 ts_tablet_manager.cc:1939] T 42d11bb37b964efd8f08e811def08318 P 84d0189b4c7342218cf6f788923580d4: Deleting consensus metadata
I20260929 16:41:32.089540 2520 catalog_manager.cc:4841] TS 84d0189b4c7342218cf6f788923580d4 (127.0.197.65:43273): tablet 42d11bb37b964efd8f08e811def08318 (table test-table [id=10c0487a1a514276b01d8894db3bc48d]) successfully deleted
I20260929 16:41:32.089834 2520 catalog_manager.cc:4841] TS 45f0267549da47abb6a5ee35c3509a9a (127.0.197.66:43603): tablet 42d11bb37b964efd8f08e811def08318 (table test-table [id=10c0487a1a514276b01d8894db3bc48d]) successfully deleted
I20260929 16:41:32.090142 3132 ts_tablet_manager.cc:1918] T 42d11bb37b964efd8f08e811def08318 P c8d5e1e737d147d5b6d47fe948fc3623: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 16:41:32.090296 3132 log.cc:1196] T 42d11bb37b964efd8f08e811def08318 P c8d5e1e737d147d5b6d47fe948fc3623: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-2/wal/wals/42d11bb37b964efd8f08e811def08318
I20260929 16:41:32.090687 3132 ts_tablet_manager.cc:1939] T 42d11bb37b964efd8f08e811def08318 P c8d5e1e737d147d5b6d47fe948fc3623: Deleting consensus metadata
I20260929 16:41:32.092024 2522 catalog_manager.cc:4841] TS c8d5e1e737d147d5b6d47fe948fc3623 (127.0.197.67:46267): tablet 42d11bb37b964efd8f08e811def08318 (table test-table [id=10c0487a1a514276b01d8894db3bc48d]) successfully deleted
Sep 29 16:41:32 dist-test-slave-l31f krb5kdc[2490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700092, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 16:41:32 dist-test-slave-l31f krb5kdc[2490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700092, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:41:32.134202 3141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:32.112974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45000 (local address 127.0.197.126:45257)
0929 16:41:32.113598 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:32.113602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:32.113626 (+ 24us) server_negotiation.cc:407] Connection header received
0929 16:41:32.113794 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:32.113798 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:32.113859 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:32.113983 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:32.115879 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:32.116718 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:32.118025 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:32.118430 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:32.118532 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:32.128476 (+ 9944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:32.128482 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:32.128487 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:32.128536 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:32.131296 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:32.132094 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:32.132098 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:32.132101 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:32.132232 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:32.132697 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:32.132702 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:32.132705 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:32.133016 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:32.133220 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:32.133582 (+ 362us) server_negotiation.cc:299] Negotiation successful
0929 16:41:32.133767 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":347,"threads_started":1}
I20260929 16:41:32.138269 2534 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45000:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:41:32.138936 2534 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:41:32.146495 2672 tablet_service.cc:1467] Processing CreateTablet for tablet abf68543714c4400860d5e942e389525 (DEFAULT_TABLE table=test-table [id=536539b7948046778d4e567c14d29578]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:41:32.146548 2812 tablet_service.cc:1467] Processing CreateTablet for tablet abf68543714c4400860d5e942e389525 (DEFAULT_TABLE table=test-table [id=536539b7948046778d4e567c14d29578]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:41:32.146600 2952 tablet_service.cc:1467] Processing CreateTablet for tablet abf68543714c4400860d5e942e389525 (DEFAULT_TABLE table=test-table [id=536539b7948046778d4e567c14d29578]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:41:32.146984 2812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abf68543714c4400860d5e942e389525. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:41:32.146983 2952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abf68543714c4400860d5e942e389525. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:41:32.146983 2672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abf68543714c4400860d5e942e389525. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:41:32.159150 3143 tablet_bootstrap.cc:492] T abf68543714c4400860d5e942e389525 P 84d0189b4c7342218cf6f788923580d4: Bootstrap starting.
I20260929 16:41:32.159741 3144 tablet_bootstrap.cc:492] T abf68543714c4400860d5e942e389525 P 45f0267549da47abb6a5ee35c3509a9a: Bootstrap starting.
I20260929 16:41:32.161103 3143 tablet_bootstrap.cc:654] T abf68543714c4400860d5e942e389525 P 84d0189b4c7342218cf6f788923580d4: Neither blocks nor log segments found. Creating new log.
I20260929 16:41:32.162127 3144 tablet_bootstrap.cc:654] T abf68543714c4400860d5e942e389525 P 45f0267549da47abb6a5ee35c3509a9a: Neither blocks nor log segments found. Creating new log.
I20260929 16:41:32.163769 3142 tablet_bootstrap.cc:492] T abf68543714c4400860d5e942e389525 P c8d5e1e737d147d5b6d47fe948fc3623: Bootstrap starting.
I20260929 16:41:32.166240 3142 tablet_bootstrap.cc:654] T abf68543714c4400860d5e942e389525 P c8d5e1e737d147d5b6d47fe948fc3623: Neither blocks nor log segments found. Creating new log.
I20260929 16:41:32.170843 3144 tablet_bootstrap.cc:492] T abf68543714c4400860d5e942e389525 P 45f0267549da47abb6a5ee35c3509a9a: No bootstrap required, opened a new log
I20260929 16:41:32.171072 3144 ts_tablet_manager.cc:1397] T abf68543714c4400860d5e942e389525 P 45f0267549da47abb6a5ee35c3509a9a: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20260929 16:41:32.171815 3143 tablet_bootstrap.cc:492] T abf68543714c4400860d5e942e389525 P 84d0189b4c7342218cf6f788923580d4: No bootstrap required, opened a new log
I20260929 16:41:32.171830 3142 tablet_bootstrap.cc:492] T abf68543714c4400860d5e942e389525 P c8d5e1e737d147d5b6d47fe948fc3623: No bootstrap required, opened a new log
I20260929 16:41:32.171975 3142 ts_tablet_manager.cc:1397] T abf68543714c4400860d5e942e389525 P c8d5e1e737d147d5b6d47fe948fc3623: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260929 16:41:32.171959 3144 raft_consensus.cc:348] T abf68543714c4400860d5e942e389525 P 45f0267549da47abb6a5ee35c3509a9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } }
I20260929 16:41:32.172230 3144 raft_consensus.cc:374] T abf68543714c4400860d5e942e389525 P 45f0267549da47abb6a5ee35c3509a9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:41:32.172158 3143 ts_tablet_manager.cc:1397] T abf68543714c4400860d5e942e389525 P 84d0189b4c7342218cf6f788923580d4: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.008s
I20260929 16:41:32.172329 3144 raft_consensus.cc:729] T abf68543714c4400860d5e942e389525 P 45f0267549da47abb6a5ee35c3509a9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45f0267549da47abb6a5ee35c3509a9a, State: Initialized, Role: FOLLOWER
I20260929 16:41:32.172631 3144 consensus_queue.cc:260] T abf68543714c4400860d5e942e389525 P 45f0267549da47abb6a5ee35c3509a9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } }
I20260929 16:41:32.172751 3142 raft_consensus.cc:348] T abf68543714c4400860d5e942e389525 P c8d5e1e737d147d5b6d47fe948fc3623 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } }
I20260929 16:41:32.172993 3142 raft_consensus.cc:374] T abf68543714c4400860d5e942e389525 P c8d5e1e737d147d5b6d47fe948fc3623 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:41:32.173089 3142 raft_consensus.cc:729] T abf68543714c4400860d5e942e389525 P c8d5e1e737d147d5b6d47fe948fc3623 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d5e1e737d147d5b6d47fe948fc3623, State: Initialized, Role: FOLLOWER
I20260929 16:41:32.173364 3142 consensus_queue.cc:260] T abf68543714c4400860d5e942e389525 P c8d5e1e737d147d5b6d47fe948fc3623 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } }
I20260929 16:41:32.173666 3143 raft_consensus.cc:348] T abf68543714c4400860d5e942e389525 P 84d0189b4c7342218cf6f788923580d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } }
I20260929 16:41:32.174050 3143 raft_consensus.cc:374] T abf68543714c4400860d5e942e389525 P 84d0189b4c7342218cf6f788923580d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:41:32.174211 3143 raft_consensus.cc:729] T abf68543714c4400860d5e942e389525 P 84d0189b4c7342218cf6f788923580d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84d0189b4c7342218cf6f788923580d4, State: Initialized, Role: FOLLOWER
I20260929 16:41:32.174623 3143 consensus_queue.cc:260] T abf68543714c4400860d5e942e389525 P 84d0189b4c7342218cf6f788923580d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } }
I20260929 16:41:32.175501 3143 ts_tablet_manager.cc:1428] T abf68543714c4400860d5e942e389525 P 84d0189b4c7342218cf6f788923580d4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 16:41:32.178433 3142 ts_tablet_manager.cc:1428] T abf68543714c4400860d5e942e389525 P c8d5e1e737d147d5b6d47fe948fc3623: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260929 16:41:32.181418 3144 ts_tablet_manager.cc:1428] T abf68543714c4400860d5e942e389525 P 45f0267549da47abb6a5ee35c3509a9a: Time spent starting tablet: real 0.010s user 0.007s sys 0.002s
I20260929 16:41:32.233341 3148 raft_consensus.cc:482] T abf68543714c4400860d5e942e389525 P 45f0267549da47abb6a5ee35c3509a9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:41:32.233636 3148 raft_consensus.cc:504] T abf68543714c4400860d5e942e389525 P 45f0267549da47abb6a5ee35c3509a9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } }
I20260929 16:41:32.234462 3148 leader_election.cc:288] T abf68543714c4400860d5e942e389525 P 45f0267549da47abb6a5ee35c3509a9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84d0189b4c7342218cf6f788923580d4 (127.0.197.65:43273), c8d5e1e737d147d5b6d47fe948fc3623 (127.0.197.67:46267)
I20260929 16:41:32.235549 2971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abf68543714c4400860d5e942e389525" candidate_uuid: "45f0267549da47abb6a5ee35c3509a9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" is_pre_election: true
I20260929 16:41:32.235549 2692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abf68543714c4400860d5e942e389525" candidate_uuid: "45f0267549da47abb6a5ee35c3509a9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d0189b4c7342218cf6f788923580d4" is_pre_election: true
I20260929 16:41:32.235950 2971 raft_consensus.cc:2454] T abf68543714c4400860d5e942e389525 P c8d5e1e737d147d5b6d47fe948fc3623 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45f0267549da47abb6a5ee35c3509a9a in term 0.
I20260929 16:41:32.236052 2692 raft_consensus.cc:2454] T abf68543714c4400860d5e942e389525 P 84d0189b4c7342218cf6f788923580d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45f0267549da47abb6a5ee35c3509a9a in term 0.
I20260929 16:41:32.236717 2763 leader_election.cc:302] T abf68543714c4400860d5e942e389525 P 45f0267549da47abb6a5ee35c3509a9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45f0267549da47abb6a5ee35c3509a9a, 84d0189b4c7342218cf6f788923580d4; no voters:
I20260929 16:41:32.237025 3148 raft_consensus.cc:2781] T abf68543714c4400860d5e942e389525 P 45f0267549da47abb6a5ee35c3509a9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:41:32.237130 3148 raft_consensus.cc:482] T abf68543714c4400860d5e942e389525 P 45f0267549da47abb6a5ee35c3509a9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:41:32.237195 3148 raft_consensus.cc:3037] T abf68543714c4400860d5e942e389525 P 45f0267549da47abb6a5ee35c3509a9a [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:41:32.239454 3148 raft_consensus.cc:504] T abf68543714c4400860d5e942e389525 P 45f0267549da47abb6a5ee35c3509a9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } }
I20260929 16:41:32.240137 3148 leader_election.cc:288] T abf68543714c4400860d5e942e389525 P 45f0267549da47abb6a5ee35c3509a9a [CANDIDATE]: Term 1 election: Requested vote from peers 84d0189b4c7342218cf6f788923580d4 (127.0.197.65:43273), c8d5e1e737d147d5b6d47fe948fc3623 (127.0.197.67:46267)
I20260929 16:41:32.240883 2692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abf68543714c4400860d5e942e389525" candidate_uuid: "45f0267549da47abb6a5ee35c3509a9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84d0189b4c7342218cf6f788923580d4"
I20260929 16:41:32.240875 2971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abf68543714c4400860d5e942e389525" candidate_uuid: "45f0267549da47abb6a5ee35c3509a9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d5e1e737d147d5b6d47fe948fc3623"
I20260929 16:41:32.241115 2692 raft_consensus.cc:3037] T abf68543714c4400860d5e942e389525 P 84d0189b4c7342218cf6f788923580d4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:41:32.241130 2971 raft_consensus.cc:3037] T abf68543714c4400860d5e942e389525 P c8d5e1e737d147d5b6d47fe948fc3623 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:41:32.244240 2971 raft_consensus.cc:2454] T abf68543714c4400860d5e942e389525 P c8d5e1e737d147d5b6d47fe948fc3623 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45f0267549da47abb6a5ee35c3509a9a in term 1.
I20260929 16:41:32.244240 2692 raft_consensus.cc:2454] T abf68543714c4400860d5e942e389525 P 84d0189b4c7342218cf6f788923580d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45f0267549da47abb6a5ee35c3509a9a in term 1.
I20260929 16:41:32.244876 2765 leader_election.cc:302] T abf68543714c4400860d5e942e389525 P 45f0267549da47abb6a5ee35c3509a9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 45f0267549da47abb6a5ee35c3509a9a, 84d0189b4c7342218cf6f788923580d4, c8d5e1e737d147d5b6d47fe948fc3623; no voters:
I20260929 16:41:32.245237 3148 raft_consensus.cc:2781] T abf68543714c4400860d5e942e389525 P 45f0267549da47abb6a5ee35c3509a9a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:41:32.246145 3148 raft_consensus.cc:686] T abf68543714c4400860d5e942e389525 P 45f0267549da47abb6a5ee35c3509a9a [term 1 LEADER]: Becoming Leader. State: Replica: 45f0267549da47abb6a5ee35c3509a9a, State: Running, Role: LEADER
I20260929 16:41:32.246680 3148 consensus_queue.cc:237] T abf68543714c4400860d5e942e389525 P 45f0267549da47abb6a5ee35c3509a9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } }
I20260929 16:41:32.249408 2534 catalog_manager.cc:5495] T abf68543714c4400860d5e942e389525 P 45f0267549da47abb6a5ee35c3509a9a reported cstate change: term changed from 0 to 1, leader changed from <none> to 45f0267549da47abb6a5ee35c3509a9a (127.0.197.66). New cstate: current_term: 1 leader_uuid: "45f0267549da47abb6a5ee35c3509a9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } health_report { overall_health: UNKNOWN } } }
I20260929 16:41:32.281810 3152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:32.276033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43738 (local address 127.0.197.66:43603)
0929 16:41:32.276615 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:32.276620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:32.276640 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:41:32.276702 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:32.276705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:32.276768 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:32.276858 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:41:32.277267 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:32.278089 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:32.279612 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:32.280077 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:32.280187 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:32.280904 (+ 717us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:41:32.281065 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:41:32.281139 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:32.281364 (+ 225us) server_negotiation.cc:299] Negotiation successful
0929 16:41:32.281488 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":334,"threads_started":1}
I20260929 16:41:32.291956 2971 raft_consensus.cc:1261] T abf68543714c4400860d5e942e389525 P c8d5e1e737d147d5b6d47fe948fc3623 [term 1 FOLLOWER]: Refusing update from remote peer 45f0267549da47abb6a5ee35c3509a9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:41:32.292027 2692 raft_consensus.cc:1261] T abf68543714c4400860d5e942e389525 P 84d0189b4c7342218cf6f788923580d4 [term 1 FOLLOWER]: Refusing update from remote peer 45f0267549da47abb6a5ee35c3509a9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:41:32.292590 3149 consensus_queue.cc:1035] T abf68543714c4400860d5e942e389525 P 45f0267549da47abb6a5ee35c3509a9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:41:32.293038 3148 consensus_queue.cc:1035] T abf68543714c4400860d5e942e389525 P 45f0267549da47abb6a5ee35c3509a9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:41:32.297230 3113 mvcc.cc:205] Tried to move back new op lower bound from 7334707578020519936 to 7334707577844899840. Current Snapshot: MvccSnapshot[applied={T|T < 7334707578020519936}]
I20260929 16:41:32.346452 2534 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45000:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:41:32.346730 2534 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45000:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:41:32.350221 2534 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e8c03d81d4e8437d90edd6d0e53056a5: Sending DeleteTablet for 3 replicas of tablet abf68543714c4400860d5e942e389525
I20260929 16:41:32.351380 2952 tablet_service.cc:1514] Processing DeleteTablet for tablet abf68543714c4400860d5e942e389525 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:41:32 UTC) from {username='kudu', principal='kudu/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:40104
I20260929 16:41:32.351449 2672 tablet_service.cc:1514] Processing DeleteTablet for tablet abf68543714c4400860d5e942e389525 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:41:32 UTC) from {username='kudu', principal='kudu/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:59056
I20260929 16:41:32.351904 3132 tablet_replica.cc:330] T abf68543714c4400860d5e942e389525 P c8d5e1e737d147d5b6d47fe948fc3623: stopping tablet replica
I20260929 16:41:32.352057 3131 tablet_replica.cc:330] T abf68543714c4400860d5e942e389525 P 84d0189b4c7342218cf6f788923580d4: stopping tablet replica
I20260929 16:41:32.352128 3132 raft_consensus.cc:2229] T abf68543714c4400860d5e942e389525 P c8d5e1e737d147d5b6d47fe948fc3623 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:41:32.352355 3132 raft_consensus.cc:2258] T abf68543714c4400860d5e942e389525 P c8d5e1e737d147d5b6d47fe948fc3623 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:41:32.352401 2812 tablet_service.cc:1514] Processing DeleteTablet for tablet abf68543714c4400860d5e942e389525 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:41:32 UTC) from {username='kudu', principal='kudu/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:43720
I20260929 16:41:32.352406 3131 raft_consensus.cc:2229] T abf68543714c4400860d5e942e389525 P 84d0189b4c7342218cf6f788923580d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:41:32.352684 3133 tablet_replica.cc:330] T abf68543714c4400860d5e942e389525 P 45f0267549da47abb6a5ee35c3509a9a: stopping tablet replica
I20260929 16:41:32.352782 3131 raft_consensus.cc:2258] T abf68543714c4400860d5e942e389525 P 84d0189b4c7342218cf6f788923580d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:41:32.352876 3133 raft_consensus.cc:2229] T abf68543714c4400860d5e942e389525 P 45f0267549da47abb6a5ee35c3509a9a [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:41:32.353107 2534 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45000:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 16:41:32.353471 3133 raft_consensus.cc:2258] T abf68543714c4400860d5e942e389525 P 45f0267549da47abb6a5ee35c3509a9a [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 16:41:32.353842 2534 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:41:32.354043 3131 ts_tablet_manager.cc:1905] T abf68543714c4400860d5e942e389525 P 84d0189b4c7342218cf6f788923580d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:41:32.354563 3132 ts_tablet_manager.cc:1905] T abf68543714c4400860d5e942e389525 P c8d5e1e737d147d5b6d47fe948fc3623: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:41:32.354915 3133 ts_tablet_manager.cc:1905] T abf68543714c4400860d5e942e389525 P 45f0267549da47abb6a5ee35c3509a9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:41:32.362452 3132 ts_tablet_manager.cc:1918] T abf68543714c4400860d5e942e389525 P c8d5e1e737d147d5b6d47fe948fc3623: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:41:32.362644 3132 log.cc:1196] T abf68543714c4400860d5e942e389525 P c8d5e1e737d147d5b6d47fe948fc3623: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-2/wal/wals/abf68543714c4400860d5e942e389525
I20260929 16:41:32.363274 3132 ts_tablet_manager.cc:1939] T abf68543714c4400860d5e942e389525 P c8d5e1e737d147d5b6d47fe948fc3623: Deleting consensus metadata
I20260929 16:41:32.363296 2672 tablet_service.cc:1467] Processing CreateTablet for tablet b348245c9c43406c9beead6659fec5e3 (DEFAULT_TABLE table=test-table [id=fa1f068b638f42bca0492f372693dafc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:41:32.363780 2672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b348245c9c43406c9beead6659fec5e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:41:32.364315 3133 ts_tablet_manager.cc:1918] T abf68543714c4400860d5e942e389525 P 45f0267549da47abb6a5ee35c3509a9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:41:32.364511 3133 log.cc:1196] T abf68543714c4400860d5e942e389525 P 45f0267549da47abb6a5ee35c3509a9a: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-1/wal/wals/abf68543714c4400860d5e942e389525
I20260929 16:41:32.364509 3131 ts_tablet_manager.cc:1918] T abf68543714c4400860d5e942e389525 P 84d0189b4c7342218cf6f788923580d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:41:32.364573 2522 catalog_manager.cc:4841] TS c8d5e1e737d147d5b6d47fe948fc3623 (127.0.197.67:46267): tablet abf68543714c4400860d5e942e389525 (table test-table [id=536539b7948046778d4e567c14d29578]) successfully deleted
I20260929 16:41:32.364813 3131 log.cc:1196] T abf68543714c4400860d5e942e389525 P 84d0189b4c7342218cf6f788923580d4: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-0/wal/wals/abf68543714c4400860d5e942e389525
I20260929 16:41:32.364961 3133 ts_tablet_manager.cc:1939] T abf68543714c4400860d5e942e389525 P 45f0267549da47abb6a5ee35c3509a9a: Deleting consensus metadata
I20260929 16:41:32.365329 3131 ts_tablet_manager.cc:1939] T abf68543714c4400860d5e942e389525 P 84d0189b4c7342218cf6f788923580d4: Deleting consensus metadata
I20260929 16:41:32.366389 2520 catalog_manager.cc:4841] TS 84d0189b4c7342218cf6f788923580d4 (127.0.197.65:43273): tablet abf68543714c4400860d5e942e389525 (table test-table [id=536539b7948046778d4e567c14d29578]) successfully deleted
I20260929 16:41:32.366631 2520 catalog_manager.cc:4841] TS 45f0267549da47abb6a5ee35c3509a9a (127.0.197.66:43603): tablet abf68543714c4400860d5e942e389525 (table test-table [id=536539b7948046778d4e567c14d29578]) successfully deleted
I20260929 16:41:32.367105 2812 tablet_service.cc:1467] Processing CreateTablet for tablet b348245c9c43406c9beead6659fec5e3 (DEFAULT_TABLE table=test-table [id=fa1f068b638f42bca0492f372693dafc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:41:32.367156 2952 tablet_service.cc:1467] Processing CreateTablet for tablet b348245c9c43406c9beead6659fec5e3 (DEFAULT_TABLE table=test-table [id=fa1f068b638f42bca0492f372693dafc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:41:32.367566 2812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b348245c9c43406c9beead6659fec5e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:41:32.367599 2952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b348245c9c43406c9beead6659fec5e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:41:32.370872 3143 tablet_bootstrap.cc:492] T b348245c9c43406c9beead6659fec5e3 P 84d0189b4c7342218cf6f788923580d4: Bootstrap starting.
I20260929 16:41:32.372638 3143 tablet_bootstrap.cc:654] T b348245c9c43406c9beead6659fec5e3 P 84d0189b4c7342218cf6f788923580d4: Neither blocks nor log segments found. Creating new log.
I20260929 16:41:32.373135 3142 tablet_bootstrap.cc:492] T b348245c9c43406c9beead6659fec5e3 P c8d5e1e737d147d5b6d47fe948fc3623: Bootstrap starting.
I20260929 16:41:32.373246 3144 tablet_bootstrap.cc:492] T b348245c9c43406c9beead6659fec5e3 P 45f0267549da47abb6a5ee35c3509a9a: Bootstrap starting.
I20260929 16:41:32.374830 3142 tablet_bootstrap.cc:654] T b348245c9c43406c9beead6659fec5e3 P c8d5e1e737d147d5b6d47fe948fc3623: Neither blocks nor log segments found. Creating new log.
I20260929 16:41:32.374830 3144 tablet_bootstrap.cc:654] T b348245c9c43406c9beead6659fec5e3 P 45f0267549da47abb6a5ee35c3509a9a: Neither blocks nor log segments found. Creating new log.
I20260929 16:41:32.375381 3143 tablet_bootstrap.cc:492] T b348245c9c43406c9beead6659fec5e3 P 84d0189b4c7342218cf6f788923580d4: No bootstrap required, opened a new log
I20260929 16:41:32.375590 3143 ts_tablet_manager.cc:1397] T b348245c9c43406c9beead6659fec5e3 P 84d0189b4c7342218cf6f788923580d4: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 16:41:32.376422 3143 raft_consensus.cc:348] T b348245c9c43406c9beead6659fec5e3 P 84d0189b4c7342218cf6f788923580d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } }
I20260929 16:41:32.376669 3143 raft_consensus.cc:374] T b348245c9c43406c9beead6659fec5e3 P 84d0189b4c7342218cf6f788923580d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:41:32.376780 3143 raft_consensus.cc:729] T b348245c9c43406c9beead6659fec5e3 P 84d0189b4c7342218cf6f788923580d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84d0189b4c7342218cf6f788923580d4, State: Initialized, Role: FOLLOWER
I20260929 16:41:32.377060 3143 consensus_queue.cc:260] T b348245c9c43406c9beead6659fec5e3 P 84d0189b4c7342218cf6f788923580d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } }
I20260929 16:41:32.377583 3143 ts_tablet_manager.cc:1428] T b348245c9c43406c9beead6659fec5e3 P 84d0189b4c7342218cf6f788923580d4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 16:41:32.380009 3142 tablet_bootstrap.cc:492] T b348245c9c43406c9beead6659fec5e3 P c8d5e1e737d147d5b6d47fe948fc3623: No bootstrap required, opened a new log
I20260929 16:41:32.380069 3144 tablet_bootstrap.cc:492] T b348245c9c43406c9beead6659fec5e3 P 45f0267549da47abb6a5ee35c3509a9a: No bootstrap required, opened a new log
I20260929 16:41:32.380203 3142 ts_tablet_manager.cc:1397] T b348245c9c43406c9beead6659fec5e3 P c8d5e1e737d147d5b6d47fe948fc3623: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260929 16:41:32.380240 3144 ts_tablet_manager.cc:1397] T b348245c9c43406c9beead6659fec5e3 P 45f0267549da47abb6a5ee35c3509a9a: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260929 16:41:32.381313 3142 raft_consensus.cc:348] T b348245c9c43406c9beead6659fec5e3 P c8d5e1e737d147d5b6d47fe948fc3623 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } }
I20260929 16:41:32.381561 3142 raft_consensus.cc:374] T b348245c9c43406c9beead6659fec5e3 P c8d5e1e737d147d5b6d47fe948fc3623 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:41:32.381628 3142 raft_consensus.cc:729] T b348245c9c43406c9beead6659fec5e3 P c8d5e1e737d147d5b6d47fe948fc3623 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d5e1e737d147d5b6d47fe948fc3623, State: Initialized, Role: FOLLOWER
I20260929 16:41:32.381049 3144 raft_consensus.cc:348] T b348245c9c43406c9beead6659fec5e3 P 45f0267549da47abb6a5ee35c3509a9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } }
I20260929 16:41:32.382504 3144 raft_consensus.cc:374] T b348245c9c43406c9beead6659fec5e3 P 45f0267549da47abb6a5ee35c3509a9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:41:32.382591 3144 raft_consensus.cc:729] T b348245c9c43406c9beead6659fec5e3 P 45f0267549da47abb6a5ee35c3509a9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45f0267549da47abb6a5ee35c3509a9a, State: Initialized, Role: FOLLOWER
I20260929 16:41:32.382529 3142 consensus_queue.cc:260] T b348245c9c43406c9beead6659fec5e3 P c8d5e1e737d147d5b6d47fe948fc3623 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } }
I20260929 16:41:32.382910 3144 consensus_queue.cc:260] T b348245c9c43406c9beead6659fec5e3 P 45f0267549da47abb6a5ee35c3509a9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } }
I20260929 16:41:32.383432 3144 ts_tablet_manager.cc:1428] T b348245c9c43406c9beead6659fec5e3 P 45f0267549da47abb6a5ee35c3509a9a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 16:41:32.384732 3142 ts_tablet_manager.cc:1428] T b348245c9c43406c9beead6659fec5e3 P c8d5e1e737d147d5b6d47fe948fc3623: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260929 16:41:32.517880 3124 raft_consensus.cc:482] T b348245c9c43406c9beead6659fec5e3 P 84d0189b4c7342218cf6f788923580d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:41:32.518098 3124 raft_consensus.cc:504] T b348245c9c43406c9beead6659fec5e3 P 84d0189b4c7342218cf6f788923580d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } }
I20260929 16:41:32.518816 3124 leader_election.cc:288] T b348245c9c43406c9beead6659fec5e3 P 84d0189b4c7342218cf6f788923580d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45f0267549da47abb6a5ee35c3509a9a (127.0.197.66:43603), c8d5e1e737d147d5b6d47fe948fc3623 (127.0.197.67:46267)
I20260929 16:41:32.519435 2836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b348245c9c43406c9beead6659fec5e3" candidate_uuid: "84d0189b4c7342218cf6f788923580d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45f0267549da47abb6a5ee35c3509a9a" is_pre_election: true
I20260929 16:41:32.519487 2971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b348245c9c43406c9beead6659fec5e3" candidate_uuid: "84d0189b4c7342218cf6f788923580d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" is_pre_election: true
I20260929 16:41:32.519687 2836 raft_consensus.cc:2454] T b348245c9c43406c9beead6659fec5e3 P 45f0267549da47abb6a5ee35c3509a9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84d0189b4c7342218cf6f788923580d4 in term 0.
I20260929 16:41:32.519795 2971 raft_consensus.cc:2454] T b348245c9c43406c9beead6659fec5e3 P c8d5e1e737d147d5b6d47fe948fc3623 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84d0189b4c7342218cf6f788923580d4 in term 0.
I20260929 16:41:32.520291 2625 leader_election.cc:302] T b348245c9c43406c9beead6659fec5e3 P 84d0189b4c7342218cf6f788923580d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84d0189b4c7342218cf6f788923580d4, c8d5e1e737d147d5b6d47fe948fc3623; no voters:
I20260929 16:41:32.520782 3124 raft_consensus.cc:2781] T b348245c9c43406c9beead6659fec5e3 P 84d0189b4c7342218cf6f788923580d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:41:32.520905 3124 raft_consensus.cc:482] T b348245c9c43406c9beead6659fec5e3 P 84d0189b4c7342218cf6f788923580d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:41:32.521005 3124 raft_consensus.cc:3037] T b348245c9c43406c9beead6659fec5e3 P 84d0189b4c7342218cf6f788923580d4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:41:32.523331 3124 raft_consensus.cc:504] T b348245c9c43406c9beead6659fec5e3 P 84d0189b4c7342218cf6f788923580d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } }
I20260929 16:41:32.523969 3124 leader_election.cc:288] T b348245c9c43406c9beead6659fec5e3 P 84d0189b4c7342218cf6f788923580d4 [CANDIDATE]: Term 1 election: Requested vote from peers 45f0267549da47abb6a5ee35c3509a9a (127.0.197.66:43603), c8d5e1e737d147d5b6d47fe948fc3623 (127.0.197.67:46267)
I20260929 16:41:32.524665 2971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b348245c9c43406c9beead6659fec5e3" candidate_uuid: "84d0189b4c7342218cf6f788923580d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d5e1e737d147d5b6d47fe948fc3623"
I20260929 16:41:32.524665 2836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b348245c9c43406c9beead6659fec5e3" candidate_uuid: "84d0189b4c7342218cf6f788923580d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45f0267549da47abb6a5ee35c3509a9a"
I20260929 16:41:32.524923 2971 raft_consensus.cc:3037] T b348245c9c43406c9beead6659fec5e3 P c8d5e1e737d147d5b6d47fe948fc3623 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:41:32.524958 2836 raft_consensus.cc:3037] T b348245c9c43406c9beead6659fec5e3 P 45f0267549da47abb6a5ee35c3509a9a [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:41:32.528036 2971 raft_consensus.cc:2454] T b348245c9c43406c9beead6659fec5e3 P c8d5e1e737d147d5b6d47fe948fc3623 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84d0189b4c7342218cf6f788923580d4 in term 1.
I20260929 16:41:32.528036 2836 raft_consensus.cc:2454] T b348245c9c43406c9beead6659fec5e3 P 45f0267549da47abb6a5ee35c3509a9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84d0189b4c7342218cf6f788923580d4 in term 1.
I20260929 16:41:32.528611 2623 leader_election.cc:302] T b348245c9c43406c9beead6659fec5e3 P 84d0189b4c7342218cf6f788923580d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 45f0267549da47abb6a5ee35c3509a9a, 84d0189b4c7342218cf6f788923580d4, c8d5e1e737d147d5b6d47fe948fc3623; no voters:
I20260929 16:41:32.528993 3124 raft_consensus.cc:2781] T b348245c9c43406c9beead6659fec5e3 P 84d0189b4c7342218cf6f788923580d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:41:32.529175 3124 raft_consensus.cc:686] T b348245c9c43406c9beead6659fec5e3 P 84d0189b4c7342218cf6f788923580d4 [term 1 LEADER]: Becoming Leader. State: Replica: 84d0189b4c7342218cf6f788923580d4, State: Running, Role: LEADER
I20260929 16:41:32.529464 3124 consensus_queue.cc:237] T b348245c9c43406c9beead6659fec5e3 P 84d0189b4c7342218cf6f788923580d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } }
I20260929 16:41:32.532815 2534 catalog_manager.cc:5495] T b348245c9c43406c9beead6659fec5e3 P 84d0189b4c7342218cf6f788923580d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84d0189b4c7342218cf6f788923580d4 (127.0.197.65). New cstate: current_term: 1 leader_uuid: "84d0189b4c7342218cf6f788923580d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84d0189b4c7342218cf6f788923580d4" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 } health_report { overall_health: UNKNOWN } } }
I20260929 16:41:32.590122 3162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:32.585343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59074 (local address 127.0.197.65:43273)
0929 16:41:32.585820 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:32.585824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:32.585841 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:41:32.585896 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:32.585900 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:32.585960 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:32.586046 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:41:32.586434 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:32.587191 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:32.588481 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:32.588888 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:32.588990 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:32.589361 (+ 371us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:41:32.589460 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:41:32.589541 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:32.589652 (+ 111us) server_negotiation.cc:299] Negotiation successful
0929 16:41:32.589716 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":283,"threads_started":1}
I20260929 16:41:32.600169 2971 raft_consensus.cc:1261] T b348245c9c43406c9beead6659fec5e3 P c8d5e1e737d147d5b6d47fe948fc3623 [term 1 FOLLOWER]: Refusing update from remote peer 84d0189b4c7342218cf6f788923580d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:41:32.600296 2836 raft_consensus.cc:1261] T b348245c9c43406c9beead6659fec5e3 P 45f0267549da47abb6a5ee35c3509a9a [term 1 FOLLOWER]: Refusing update from remote peer 84d0189b4c7342218cf6f788923580d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:41:32.600841 3124 consensus_queue.cc:1035] T b348245c9c43406c9beead6659fec5e3 P 84d0189b4c7342218cf6f788923580d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d5e1e737d147d5b6d47fe948fc3623" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:41:32.601171 3123 consensus_queue.cc:1035] T b348245c9c43406c9beead6659fec5e3 P 84d0189b4c7342218cf6f788923580d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45f0267549da47abb6a5ee35c3509a9a" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 16:41:32.623760 3114 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 16:41:34.222095 2534 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45000:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:41:34.222318 2534 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45000:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:41:34.227047 2534 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e8c03d81d4e8437d90edd6d0e53056a5: Sending DeleteTablet for 3 replicas of tablet b348245c9c43406c9beead6659fec5e3
I20260929 16:41:34.228073 2672 tablet_service.cc:1514] Processing DeleteTablet for tablet b348245c9c43406c9beead6659fec5e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:41:34 UTC) from {username='kudu', principal='kudu/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:59056
I20260929 16:41:34.228123 2812 tablet_service.cc:1514] Processing DeleteTablet for tablet b348245c9c43406c9beead6659fec5e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:41:34 UTC) from {username='kudu', principal='kudu/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:43720
I20260929 16:41:34.228477 2952 tablet_service.cc:1514] Processing DeleteTablet for tablet b348245c9c43406c9beead6659fec5e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:41:34 UTC) from {username='kudu', principal='kudu/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:40104
I20260929 16:41:34.228649 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 2606
I20260929 16:41:34.238715 3191 tablet_replica.cc:330] T b348245c9c43406c9beead6659fec5e3 P c8d5e1e737d147d5b6d47fe948fc3623: stopping tablet replica
I20260929 16:41:34.239926 3191 raft_consensus.cc:2229] T b348245c9c43406c9beead6659fec5e3 P c8d5e1e737d147d5b6d47fe948fc3623 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:41:34.240144 3190 tablet_replica.cc:330] T b348245c9c43406c9beead6659fec5e3 P 84d0189b4c7342218cf6f788923580d4: stopping tablet replica
I20260929 16:41:34.240486 3191 raft_consensus.cc:2258] T b348245c9c43406c9beead6659fec5e3 P c8d5e1e737d147d5b6d47fe948fc3623 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:41:34.240655 3189 tablet_replica.cc:330] T b348245c9c43406c9beead6659fec5e3 P 45f0267549da47abb6a5ee35c3509a9a: stopping tablet replica
I20260929 16:41:34.240878 3189 raft_consensus.cc:2229] T b348245c9c43406c9beead6659fec5e3 P 45f0267549da47abb6a5ee35c3509a9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:41:34.240909 3190 raft_consensus.cc:2229] T b348245c9c43406c9beead6659fec5e3 P 84d0189b4c7342218cf6f788923580d4 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:41:34.241107 3189 raft_consensus.cc:2258] T b348245c9c43406c9beead6659fec5e3 P 45f0267549da47abb6a5ee35c3509a9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:41:34.241611 3190 raft_consensus.cc:2258] T b348245c9c43406c9beead6659fec5e3 P 84d0189b4c7342218cf6f788923580d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:41:34.242054 3191 ts_tablet_manager.cc:1905] T b348245c9c43406c9beead6659fec5e3 P c8d5e1e737d147d5b6d47fe948fc3623: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:41:34.242067 3189 ts_tablet_manager.cc:1905] T b348245c9c43406c9beead6659fec5e3 P 45f0267549da47abb6a5ee35c3509a9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:41:34.242707 3190 ts_tablet_manager.cc:1905] T b348245c9c43406c9beead6659fec5e3 P 84d0189b4c7342218cf6f788923580d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:41:34.249330 3189 ts_tablet_manager.cc:1918] T b348245c9c43406c9beead6659fec5e3 P 45f0267549da47abb6a5ee35c3509a9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 16:41:34.249501 3189 log.cc:1196] T b348245c9c43406c9beead6659fec5e3 P 45f0267549da47abb6a5ee35c3509a9a: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-1/wal/wals/b348245c9c43406c9beead6659fec5e3
I20260929 16:41:34.249990 3189 ts_tablet_manager.cc:1939] T b348245c9c43406c9beead6659fec5e3 P 45f0267549da47abb6a5ee35c3509a9a: Deleting consensus metadata
I20260929 16:41:34.250236 3191 ts_tablet_manager.cc:1918] T b348245c9c43406c9beead6659fec5e3 P c8d5e1e737d147d5b6d47fe948fc3623: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 16:41:34.250385 3191 log.cc:1196] T b348245c9c43406c9beead6659fec5e3 P c8d5e1e737d147d5b6d47fe948fc3623: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-2/wal/wals/b348245c9c43406c9beead6659fec5e3
Sep 29 16:41:34 dist-test-slave-l31f krb5kdc[2490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700092, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.197.65@KRBTEST.COM
I20260929 16:41:34.250900 3191 ts_tablet_manager.cc:1939] T b348245c9c43406c9beead6659fec5e3 P c8d5e1e737d147d5b6d47fe948fc3623: Deleting consensus metadata
I20260929 16:41:34.251148 3190 ts_tablet_manager.cc:1918] T b348245c9c43406c9beead6659fec5e3 P 84d0189b4c7342218cf6f788923580d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 16:41:34.251385 3190 log.cc:1196] T b348245c9c43406c9beead6659fec5e3 P 84d0189b4c7342218cf6f788923580d4: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790700068255611-789-0/minicluster-data/ts-0/wal/wals/b348245c9c43406c9beead6659fec5e3
I20260929 16:41:34.251379 2520 catalog_manager.cc:4841] TS 45f0267549da47abb6a5ee35c3509a9a (127.0.197.66:43603): tablet b348245c9c43406c9beead6659fec5e3 (table test-table [id=fa1f068b638f42bca0492f372693dafc]) successfully deleted
I20260929 16:41:34.251962 3190 ts_tablet_manager.cc:1939] T b348245c9c43406c9beead6659fec5e3 P 84d0189b4c7342218cf6f788923580d4: Deleting consensus metadata
I20260929 16:41:34.252054 2522 catalog_manager.cc:4841] TS c8d5e1e737d147d5b6d47fe948fc3623 (127.0.197.67:46267): tablet b348245c9c43406c9beead6659fec5e3 (table test-table [id=fa1f068b638f42bca0492f372693dafc]) successfully deleted
I20260929 16:41:34.253000 2520 catalog_manager.cc:4841] TS 84d0189b4c7342218cf6f788923580d4 (127.0.197.65:43273): tablet b348245c9c43406c9beead6659fec5e3 (table test-table [id=fa1f068b638f42bca0492f372693dafc]) successfully deleted
I20260929 16:41:34.258539 3192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:34.231283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49204 (local address 127.0.197.65:43273)
0929 16:41:34.238946 (+ 7663us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:34.238956 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:34.239532 (+ 576us) server_negotiation.cc:407] Connection header received
0929 16:41:34.239759 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:34.239767 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:34.239883 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:34.240028 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:34.242006 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:34.243168 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:34.245098 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:34.245801 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:34.245978 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:34.251989 (+ 6011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:34.252011 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:34.252035 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:34.252086 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:34.255543 (+ 3457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:34.256349 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:34.256361 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:34.256365 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:34.256537 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:34.256940 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:34.256947 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:34.256950 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:34.257302 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:34.257625 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:34.257919 (+ 294us) server_negotiation.cc:299] Negotiation successful
0929 16:41:34.258153 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7458,"thread_start_us":7323,"threads_started":1}
W20260929 16:41:34.259363 2737 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49204
W20260929 16:41:34.260092 789 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 2606: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:41:34.260375 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 2606
I20260929 16:41:34.283600 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 2746
Sep 29 16:41:34 dist-test-slave-l31f krb5kdc[2490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700092, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.197.66@KRBTEST.COM
I20260929 16:41:34.300956 3194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:34.284816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48730 (local address 127.0.197.66:43603)
0929 16:41:34.285746 (+ 930us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:34.285751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:34.285773 (+ 22us) server_negotiation.cc:407] Connection header received
0929 16:41:34.285863 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:34.285867 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:34.285988 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:34.286203 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:34.287711 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:34.288466 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:34.289721 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:34.290226 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:34.290371 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:34.294756 (+ 4385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:34.294772 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:34.294790 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:34.294845 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:34.298116 (+ 3271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:34.298893 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:34.298900 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:34.298904 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:34.299143 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:34.299468 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:34.299471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:34.299473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:34.299822 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:34.300055 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:34.300359 (+ 304us) server_negotiation.cc:299] Negotiation successful
0929 16:41:34.300555 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":737,"thread_start_us":586,"threads_started":1}
W20260929 16:41:34.301635 2877 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48730
W20260929 16:41:34.302364 789 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 2746: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:41:34.302497 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 2746
I20260929 16:41:34.329861 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 2886
Sep 29 16:41:34 dist-test-slave-l31f krb5kdc[2490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700092, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.197.67@KRBTEST.COM
I20260929 16:41:34.349329 3195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:34.330789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36754 (local address 127.0.197.67:46267)
0929 16:41:34.331421 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:34.331426 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:34.331445 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:41:34.331514 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:34.331518 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:34.331582 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:34.331679 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:34.333367 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:34.334137 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:34.336420 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:34.336879 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:34.337003 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:34.342450 (+ 5447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:34.342467 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:34.342484 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:34.342527 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:34.346192 (+ 3665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:34.347099 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:34.347107 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:34.347109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:34.347252 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:34.347699 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:34.347704 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:34.347706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:34.348080 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:34.348386 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:34.348703 (+ 317us) server_negotiation.cc:299] Negotiation successful
0929 16:41:34.348940 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":404,"threads_started":1}
W20260929 16:41:34.350099 3017 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36754
W20260929 16:41:34.350835 789 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 2886: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:41:34.350986 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 2886
I20260929 16:41:34.372941 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 2503
I20260929 16:41:34.374150 2575 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:41:34.493659 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 2503
2026-09-29T16:41:34Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8987 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:41:34 dist-test-slave-l31f krb5kdc[3199](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:41:34 dist-test-slave-l31f krb5kdc[3199](info): set up 2 sockets
Sep 29 16:41:34 dist-test-slave-l31f krb5kdc[3199](info): commencing operation
krb5kdc: starting...
W20260929 16:41:35.623857 789 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/krb5kdc/test-user.keytab.
Sep 29 16:41:35 dist-test-slave-l31f krb5kdc[3199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:41:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:41:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.197.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
I20260929 16:41:35.841004 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:40831
--webserver_interface=127.0.197.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:41587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.126:40831
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:41:36.195223 3212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:41:36.195555 3212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:41:36.195717 3212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:41:36.203491 3212 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:41:36.203631 3212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:41:36.203679 3212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:41:36.203811 3212 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:41:36.203927 3212 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:41:36.214299 3212 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:41587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.197.126:40831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:40831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.197.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:41:36.215799 3212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:41:36.217374 3212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:41:36.231432 3217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:36.231731 3220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:36.231421 3218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:41:36.231953 3212 server_base.cc:1034] running on GCE node
I20260929 16:41:36.233031 3212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:41:36.234655 3212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:41:36.235914 3212 hybrid_clock.cc:648] HybridClock initialized: now 1790700096235824 us; error 78 us; skew 500 ppm
I20260929 16:41:36.236342 3212 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:41:36 dist-test-slave-l31f krb5kdc[3199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700096, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:41:36.239686 3212 init.cc:377] Logged in from keytab as kudu/127.0.197.126@KRBTEST.COM (short username kudu)
I20260929 16:41:36.241448 3212 webserver.cc:460] Webserver started at http://127.0.197.126:45949/ using document root <none> and password file <none>
I20260929 16:41:36.242177 3212 fs_manager.cc:359] Metadata directory not provided
I20260929 16:41:36.242280 3212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:41:36.242532 3212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:41:36.245678 3212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/master-0/data/instance:
uuid: "ecbc049bca5842fdbeb4f2113b8ab51f"
format_stamp: "Formatted at 2026-09-29 16:41:36 on dist-test-slave-l31f"
server_key: "6970cb5207a7b99f202d97429995a43b"
server_key_iv: "6b0397533ffbe3aa9306be562aa5c29c"
server_key_version: "encryptionkey@0"
I20260929 16:41:36.246479 3212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/master-0/wal/instance:
uuid: "ecbc049bca5842fdbeb4f2113b8ab51f"
format_stamp: "Formatted at 2026-09-29 16:41:36 on dist-test-slave-l31f"
server_key: "6970cb5207a7b99f202d97429995a43b"
server_key_iv: "6b0397533ffbe3aa9306be562aa5c29c"
server_key_version: "encryptionkey@0"
I20260929 16:41:36.252307 3212 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260929 16:41:36.256531 3227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:36.258232 3212 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:41:36.258491 3212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "ecbc049bca5842fdbeb4f2113b8ab51f"
format_stamp: "Formatted at 2026-09-29 16:41:36 on dist-test-slave-l31f"
server_key: "6970cb5207a7b99f202d97429995a43b"
server_key_iv: "6b0397533ffbe3aa9306be562aa5c29c"
server_key_version: "encryptionkey@0"
I20260929 16:41:36.258755 3212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:41:36.275882 3212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:41:36.280848 3212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:41:36.281200 3212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:41:36.302098 3212 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.126:40831
I20260929 16:41:36.302103 3278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.126:40831 every 8 connection(s)
I20260929 16:41:36.303807 3212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:41:36.308116 3279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:41:36.313287 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 3212
I20260929 16:41:36.313556 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/master-0/wal/instance
I20260929 16:41:36.314369 789 external_mini_cluster.cc:1428] Setting key 435ae1782d8d93b50a07bd68b3bf8e11
I20260929 16:41:36.321861 3279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecbc049bca5842fdbeb4f2113b8ab51f: Bootstrap starting.
I20260929 16:41:36.325500 3279 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ecbc049bca5842fdbeb4f2113b8ab51f: Neither blocks nor log segments found. Creating new log.
I20260929 16:41:36.327054 3279 log.cc:824] T 00000000000000000000000000000000 P ecbc049bca5842fdbeb4f2113b8ab51f: Log is configured to *not* fsync() on all Append() calls
I20260929 16:41:36.331547 3279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecbc049bca5842fdbeb4f2113b8ab51f: No bootstrap required, opened a new log
Sep 29 16:41:36 dist-test-slave-l31f krb5kdc[3199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:41:36.343678 3279 raft_consensus.cc:348] T 00000000000000000000000000000000 P ecbc049bca5842fdbeb4f2113b8ab51f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecbc049bca5842fdbeb4f2113b8ab51f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 40831 } }
I20260929 16:41:36.344309 3279 raft_consensus.cc:374] T 00000000000000000000000000000000 P ecbc049bca5842fdbeb4f2113b8ab51f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:41:36.344427 3279 raft_consensus.cc:729] T 00000000000000000000000000000000 P ecbc049bca5842fdbeb4f2113b8ab51f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecbc049bca5842fdbeb4f2113b8ab51f, State: Initialized, Role: FOLLOWER
I20260929 16:41:36.345232 3279 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecbc049bca5842fdbeb4f2113b8ab51f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecbc049bca5842fdbeb4f2113b8ab51f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 40831 } }
I20260929 16:41:36.345578 3279 raft_consensus.cc:388] T 00000000000000000000000000000000 P ecbc049bca5842fdbeb4f2113b8ab51f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:41:36.345750 3279 raft_consensus.cc:482] T 00000000000000000000000000000000 P ecbc049bca5842fdbeb4f2113b8ab51f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:41:36.345909 3282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:36.316874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52924 (local address 127.0.197.126:40831)
0929 16:41:36.317743 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:36.317757 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:36.317800 (+ 43us) server_negotiation.cc:407] Connection header received
0929 16:41:36.320110 (+ 2310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:36.320140 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:36.320601 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:36.321051 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:36.322683 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:36.324678 (+ 1995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:36.326926 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:36.328475 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:36.328706 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:36.334918 (+ 6212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:36.334932 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:36.334961 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:36.335046 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:36.339521 (+ 4475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:36.340561 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:36.340572 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:36.340583 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:36.340850 (+ 267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:36.341339 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:36.341347 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:36.341352 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:36.342233 (+ 881us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:36.342539 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:36.343536 (+ 997us) server_negotiation.cc:299] Negotiation successful
0929 16:41:36.344236 (+ 700us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":358,"threads_started":1}
I20260929 16:41:36.345986 3279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ecbc049bca5842fdbeb4f2113b8ab51f [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:41:36.348906 3279 raft_consensus.cc:504] T 00000000000000000000000000000000 P ecbc049bca5842fdbeb4f2113b8ab51f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecbc049bca5842fdbeb4f2113b8ab51f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 40831 } }
I20260929 16:41:36.349634 3279 leader_election.cc:302] T 00000000000000000000000000000000 P ecbc049bca5842fdbeb4f2113b8ab51f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ecbc049bca5842fdbeb4f2113b8ab51f; no voters:
I20260929 16:41:36.350673 3279 leader_election.cc:288] T 00000000000000000000000000000000 P ecbc049bca5842fdbeb4f2113b8ab51f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:41:36.350874 3284 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ecbc049bca5842fdbeb4f2113b8ab51f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:41:36.352368 3284 raft_consensus.cc:686] T 00000000000000000000000000000000 P ecbc049bca5842fdbeb4f2113b8ab51f [term 1 LEADER]: Becoming Leader. State: Replica: ecbc049bca5842fdbeb4f2113b8ab51f, State: Running, Role: LEADER
I20260929 16:41:36.353502 3284 consensus_queue.cc:237] T 00000000000000000000000000000000 P ecbc049bca5842fdbeb4f2113b8ab51f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecbc049bca5842fdbeb4f2113b8ab51f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 40831 } }
I20260929 16:41:36.354833 3279 sys_catalog.cc:564] T 00000000000000000000000000000000 P ecbc049bca5842fdbeb4f2113b8ab51f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:41:36.357715 3286 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecbc049bca5842fdbeb4f2113b8ab51f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ecbc049bca5842fdbeb4f2113b8ab51f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecbc049bca5842fdbeb4f2113b8ab51f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 40831 } } }
I20260929 16:41:36.358425 3285 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecbc049bca5842fdbeb4f2113b8ab51f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ecbc049bca5842fdbeb4f2113b8ab51f. Latest consensus state: current_term: 1 leader_uuid: "ecbc049bca5842fdbeb4f2113b8ab51f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecbc049bca5842fdbeb4f2113b8ab51f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 40831 } } }
I20260929 16:41:36.359339 3286 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecbc049bca5842fdbeb4f2113b8ab51f [sys.catalog]: This master's current role is: LEADER
I20260929 16:41:36.359460 3285 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecbc049bca5842fdbeb4f2113b8ab51f [sys.catalog]: This master's current role is: LEADER
I20260929 16:41:36.370884 3291 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:41:36.379226 3291 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:41:36.389525 3291 catalog_manager.cc:1347] Generated new cluster ID: 5bd33027796d4e6da915fe5ec5b3ef1c
I20260929 16:41:36.389691 3291 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:41:36.414896 3291 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:41:36.416374 3291 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:41:36.424018 3291 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ecbc049bca5842fdbeb4f2113b8ab51f: Generated new TSK 0
I20260929 16:41:36.425266 3291 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.197.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab.
Entry for principal kudu/127.0.197.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab.
I20260929 16:41:36.484527 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.65:0
--local_ip_for_outbound_sockets=127.0.197.65
--webserver_interface=127.0.197.65
--webserver_port=0
--tserver_master_addrs=127.0.197.126:40831
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab
--principal=kudu/127.0.197.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:41587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:41:36.852953 3305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:41:36.853284 3305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:41:36.853384 3305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:41:36.861281 3305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:41:36.861421 3305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:41:36.861618 3305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.65
I20260929 16:41:36.872288 3305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:41587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab
--principal=kudu/127.0.197.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.197.65
--webserver_port=0
--tserver_master_addrs=127.0.197.126:40831
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.65
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:41:36.873725 3305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:41:36.875181 3305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:41:36.889552 3311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:36.889679 3310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:36.889642 3313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:41:36.890866 3305 server_base.cc:1034] running on GCE node
I20260929 16:41:36.891669 3305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:41:36.892849 3305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:41:36.894075 3305 hybrid_clock.cc:648] HybridClock initialized: now 1790700096894025 us; error 43 us; skew 500 ppm
I20260929 16:41:36.894446 3305 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:41:36 dist-test-slave-l31f krb5kdc[3199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700096, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:41:36.897997 3305 init.cc:377] Logged in from keytab as kudu/127.0.197.65@KRBTEST.COM (short username kudu)
I20260929 16:41:36.899891 3305 webserver.cc:460] Webserver started at http://127.0.197.65:33669/ using document root <none> and password file <none>
I20260929 16:41:36.900712 3305 fs_manager.cc:359] Metadata directory not provided
I20260929 16:41:36.900872 3305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:41:36.901248 3305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:41:36.904620 3305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-0/data/instance:
uuid: "90d398cdd83d403d84d8a6149f1ab44a"
format_stamp: "Formatted at 2026-09-29 16:41:36 on dist-test-slave-l31f"
server_key: "85d43b1997c18d706108ee3c8d81726f"
server_key_iv: "958f3e8bc8737b1d1f1be949475e4420"
server_key_version: "encryptionkey@0"
I20260929 16:41:36.905544 3305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-0/wal/instance:
uuid: "90d398cdd83d403d84d8a6149f1ab44a"
format_stamp: "Formatted at 2026-09-29 16:41:36 on dist-test-slave-l31f"
server_key: "85d43b1997c18d706108ee3c8d81726f"
server_key_iv: "958f3e8bc8737b1d1f1be949475e4420"
server_key_version: "encryptionkey@0"
I20260929 16:41:36.911659 3305 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 16:41:36.915845 3320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:36.917407 3305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20260929 16:41:36.917635 3305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-0/wal
uuid: "90d398cdd83d403d84d8a6149f1ab44a"
format_stamp: "Formatted at 2026-09-29 16:41:36 on dist-test-slave-l31f"
server_key: "85d43b1997c18d706108ee3c8d81726f"
server_key_iv: "958f3e8bc8737b1d1f1be949475e4420"
server_key_version: "encryptionkey@0"
I20260929 16:41:36.917874 3305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:41:36.970315 3305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:41:36.975075 3305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:41:36.975420 3305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:41:36.976482 3305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:41:36.977936 3305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:41:36.978060 3305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:36.978225 3305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:41:36.978298 3305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:37.029659 3305 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.65:38931
I20260929 16:41:37.029875 3432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.65:38931 every 8 connection(s)
I20260929 16:41:37.031561 3305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
I20260929 16:41:37.037693 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 3305
I20260929 16:41:37.038008 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-0/wal/instance
I20260929 16:41:37.038769 789 external_mini_cluster.cc:1428] Setting key affe1133bdeba75a4b22c416a7ab5845
Sep 29 16:41:37 dist-test-slave-l31f krb5kdc[3199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700096, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.65@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.197.66@KRBTEST.COM; defaulting to no policy
I20260929 16:41:37.058941 3435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:37.035403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:46777 (local address 127.0.197.126:40831)
0929 16:41:37.035916 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:37.035923 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:37.036982 (+ 1059us) server_negotiation.cc:407] Connection header received
0929 16:41:37.039872 (+ 2890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:37.039877 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:37.039986 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:37.040137 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:37.042125 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:37.042948 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:37.045160 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:37.045960 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:37.046137 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:37.050573 (+ 4436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:37.050580 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:37.050583 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:37.050624 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:37.054852 (+ 4228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:37.056128 (+ 1276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:37.056134 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:37.056137 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:37.056380 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:37.056881 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:37.056887 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:37.056890 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:37.057190 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:37.057359 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:37.058211 (+ 852us) server_negotiation.cc:299] Negotiation successful
0929 16:41:37.058488 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":256,"threads_started":1}
I20260929 16:41:37.060094 3436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:37.036023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:40831 (local address 127.0.197.65:46777)
0929 16:41:37.036795 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:37.036836 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:37.039477 (+ 2641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:37.040468 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:37.040479 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:37.041261 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:41:37.041917 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:37.041938 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:37.043164 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:37.043169 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:37.044931 (+ 1762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:37.044953 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:37.046273 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:37.046281 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:37.046567 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:37.047458 (+ 891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:41:37.047490 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:41:37.050247 (+ 2757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:41:37.055283 (+ 5036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:37.055298 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:37.055323 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:37.055968 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:37.056510 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:37.056515 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:37.056519 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:37.056743 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:37.057331 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:41:37.057340 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:41:37.057802 (+ 462us) client_negotiation.cc:769] Sending connection context
0929 16:41:37.058116 (+ 314us) client_negotiation.cc:240] Negotiation successful
0929 16:41:37.058748 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":297,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.66@KRBTEST.COM" created.
I20260929 16:41:37.062119 3433 heartbeater.cc:347] Connected to a master server at 127.0.197.126:40831
I20260929 16:41:37.062665 3433 heartbeater.cc:464] Registering TS with master...
I20260929 16:41:37.063918 3433 heartbeater.cc:511] Master 127.0.197.126:40831 requested a full tablet report, sending...
I20260929 16:41:37.066915 3244 ts_manager.cc:194] Registered new tserver with Master: 90d398cdd83d403d84d8a6149f1ab44a (127.0.197.65:38931)
I20260929 16:41:37.069362 3244 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.65@KRBTEST.COM'} at 127.0.197.65:46777
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab.
Entry for principal kudu/127.0.197.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab.
I20260929 16:41:37.091475 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.66:0
--local_ip_for_outbound_sockets=127.0.197.66
--webserver_interface=127.0.197.66
--webserver_port=0
--tserver_master_addrs=127.0.197.126:40831
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab
--principal=kudu/127.0.197.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:41587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:41:37.439491 3439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:41:37.439819 3439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:41:37.439917 3439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:41:37.447547 3439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:41:37.447688 3439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:41:37.447839 3439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.66
I20260929 16:41:37.458319 3439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:41587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab
--principal=kudu/127.0.197.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.197.66
--webserver_port=0
--tserver_master_addrs=127.0.197.126:40831
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.66
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:41:37.459780 3439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:41:37.461213 3439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:41:37.476537 3445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:37.476598 3447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:37.476725 3444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:41:37.477859 3439 server_base.cc:1034] running on GCE node
I20260929 16:41:37.478595 3439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:41:37.479719 3439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:41:37.480984 3439 hybrid_clock.cc:648] HybridClock initialized: now 1790700097480927 us; error 43 us; skew 500 ppm
I20260929 16:41:37.481370 3439 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:41:37 dist-test-slave-l31f krb5kdc[3199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700097, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:41:37.492233 3439 init.cc:377] Logged in from keytab as kudu/127.0.197.66@KRBTEST.COM (short username kudu)
I20260929 16:41:37.494136 3439 webserver.cc:460] Webserver started at http://127.0.197.66:35243/ using document root <none> and password file <none>
I20260929 16:41:37.494946 3439 fs_manager.cc:359] Metadata directory not provided
I20260929 16:41:37.495184 3439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:41:37.495589 3439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:41:37.498795 3439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-1/data/instance:
uuid: "9eeb66f268dd4861b3678752a469a531"
format_stamp: "Formatted at 2026-09-29 16:41:37 on dist-test-slave-l31f"
server_key: "3643614b959f327ea6d162bb5467aba4"
server_key_iv: "d5d0b64f677c454b6c20e4760565fe53"
server_key_version: "encryptionkey@0"
I20260929 16:41:37.499820 3439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-1/wal/instance:
uuid: "9eeb66f268dd4861b3678752a469a531"
format_stamp: "Formatted at 2026-09-29 16:41:37 on dist-test-slave-l31f"
server_key: "3643614b959f327ea6d162bb5467aba4"
server_key_iv: "d5d0b64f677c454b6c20e4760565fe53"
server_key_version: "encryptionkey@0"
I20260929 16:41:37.505676 3439 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260929 16:41:37.509913 3454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:37.511653 3439 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 16:41:37.511881 3439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-1/wal
uuid: "9eeb66f268dd4861b3678752a469a531"
format_stamp: "Formatted at 2026-09-29 16:41:37 on dist-test-slave-l31f"
server_key: "3643614b959f327ea6d162bb5467aba4"
server_key_iv: "d5d0b64f677c454b6c20e4760565fe53"
server_key_version: "encryptionkey@0"
I20260929 16:41:37.512084 3439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:41:37.533280 3439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:41:37.538864 3439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:41:37.539316 3439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:41:37.540395 3439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:41:37.541914 3439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:41:37.541996 3439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:37.542088 3439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:41:37.542131 3439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:37.577751 3439 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.66:44557
I20260929 16:41:37.577811 3566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.66:44557 every 8 connection(s)
I20260929 16:41:37.579191 3439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
I20260929 16:41:37.582600 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 3439
I20260929 16:41:37.582814 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-1/wal/instance
I20260929 16:41:37.583344 789 external_mini_cluster.cc:1428] Setting key 1c694b61bfb518548cfb48917e4d818e
Sep 29 16:41:37 dist-test-slave-l31f krb5kdc[3199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700097, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.66@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:41:37.601720 3569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:37.581960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:56155 (local address 127.0.197.126:40831)
0929 16:41:37.582454 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:37.582461 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:37.583614 (+ 1153us) server_negotiation.cc:407] Connection header received
0929 16:41:37.585505 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:37.585510 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:37.585590 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:37.585705 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:37.587381 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:37.588530 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:37.590269 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:37.590682 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:37.590781 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:37.595509 (+ 4728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:37.595518 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:37.595522 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:37.595570 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:37.598409 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:37.599333 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:37.599338 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:37.599340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:37.599470 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:37.599935 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:37.599944 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:37.599946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:37.600211 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:37.600365 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:37.601116 (+ 751us) server_negotiation.cc:299] Negotiation successful
0929 16:41:37.601348 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":237,"threads_started":1}
WARNING: no policy specified for kudu/127.0.197.67@KRBTEST.COM; defaulting to no policy
I20260929 16:41:37.602784 3570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:37.582515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:40831 (local address 127.0.197.66:56155)
0929 16:41:37.583420 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:37.583468 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:37.585229 (+ 1761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:37.585949 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:37.585958 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:37.586497 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:41:37.587195 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:37.587216 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:37.588718 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:37.588722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:37.590095 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:37.590115 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:37.590953 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:37.590964 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:37.591364 (+ 400us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:37.592191 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:41:37.592233 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:41:37.595154 (+ 2921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:41:37.598589 (+ 3435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:37.598600 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:37.598628 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:37.599173 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:37.599593 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:37.599598 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:37.599602 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:37.599811 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:37.600350 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:41:37.600359 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:41:37.600735 (+ 376us) client_negotiation.cc:769] Sending connection context
0929 16:41:37.601021 (+ 286us) client_negotiation.cc:240] Negotiation successful
0929 16:41:37.601610 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":420,"threads_started":1}
I20260929 16:41:37.604552 3567 heartbeater.cc:347] Connected to a master server at 127.0.197.126:40831
I20260929 16:41:37.604941 3567 heartbeater.cc:464] Registering TS with master...
I20260929 16:41:37.605821 3567 heartbeater.cc:511] Master 127.0.197.126:40831 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.67@KRBTEST.COM" created.
I20260929 16:41:37.607888 3244 ts_manager.cc:194] Registered new tserver with Master: 9eeb66f268dd4861b3678752a469a531 (127.0.197.66:44557)
I20260929 16:41:37.608786 3244 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.66@KRBTEST.COM'} at 127.0.197.66:56155
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab.
Entry for principal kudu/127.0.197.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab.
I20260929 16:41:37.637363 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.67:0
--local_ip_for_outbound_sockets=127.0.197.67
--webserver_interface=127.0.197.67
--webserver_port=0
--tserver_master_addrs=127.0.197.126:40831
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab
--principal=kudu/127.0.197.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:41587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:41:37.989944 3573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:41:37.990259 3573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:41:37.990360 3573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:41:37.998203 3573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:41:37.998349 3573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:41:37.998484 3573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.67
I20260929 16:41:38.008833 3573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:41587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab
--principal=kudu/127.0.197.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.197.67
--webserver_port=0
--tserver_master_addrs=127.0.197.126:40831
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.67
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:41:38.010290 3573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:41:38.011772 3573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:41:38.026961 3578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:38.026887 3579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:38.026912 3581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:41:38.027694 3573 server_base.cc:1034] running on GCE node
I20260929 16:41:38.028424 3573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:41:38.029567 3573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:41:38.030850 3573 hybrid_clock.cc:648] HybridClock initialized: now 1790700098030788 us; error 70 us; skew 500 ppm
I20260929 16:41:38.031260 3573 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:41:38 dist-test-slave-l31f krb5kdc[3199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:41:38.034899 3573 init.cc:377] Logged in from keytab as kudu/127.0.197.67@KRBTEST.COM (short username kudu)
I20260929 16:41:38.036736 3573 webserver.cc:460] Webserver started at http://127.0.197.67:43965/ using document root <none> and password file <none>
I20260929 16:41:38.037532 3573 fs_manager.cc:359] Metadata directory not provided
I20260929 16:41:38.037695 3573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:41:38.038041 3573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:41:38.041273 3573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-2/data/instance:
uuid: "575e23ee64034d40b3e93312c1d713c0"
format_stamp: "Formatted at 2026-09-29 16:41:38 on dist-test-slave-l31f"
server_key: "bd14b991c2a1d8d3fba82e84d7c68540"
server_key_iv: "c77862f84a7c436bdc3dcfc6883c84c5"
server_key_version: "encryptionkey@0"
I20260929 16:41:38.042176 3573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-2/wal/instance:
uuid: "575e23ee64034d40b3e93312c1d713c0"
format_stamp: "Formatted at 2026-09-29 16:41:38 on dist-test-slave-l31f"
server_key: "bd14b991c2a1d8d3fba82e84d7c68540"
server_key_iv: "c77862f84a7c436bdc3dcfc6883c84c5"
server_key_version: "encryptionkey@0"
I20260929 16:41:38.048064 3573 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260929 16:41:38.052044 3588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:38.053691 3573 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260929 16:41:38.053938 3573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-2/wal
uuid: "575e23ee64034d40b3e93312c1d713c0"
format_stamp: "Formatted at 2026-09-29 16:41:38 on dist-test-slave-l31f"
server_key: "bd14b991c2a1d8d3fba82e84d7c68540"
server_key_iv: "c77862f84a7c436bdc3dcfc6883c84c5"
server_key_version: "encryptionkey@0"
I20260929 16:41:38.054194 3573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:41:38.073412 3433 heartbeater.cc:503] Master 127.0.197.126:40831 was elected leader, sending a full tablet report...
I20260929 16:41:38.074252 3573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:41:38.079342 3573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:41:38.079667 3573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:41:38.080734 3573 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:41:38.082199 3573 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:41:38.082274 3573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:38.082450 3573 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:41:38.082511 3573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:38.117096 3573 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.67:36421
I20260929 16:41:38.117146 3700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.67:36421 every 8 connection(s)
I20260929 16:41:38.118673 3573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
I20260929 16:41:38.128659 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 3573
I20260929 16:41:38.128887 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-2/wal/instance
I20260929 16:41:38.129400 789 external_mini_cluster.cc:1428] Setting key 973e93bbe88bf2f9d18204aefdecaf6a
Sep 29 16:41:38 dist-test-slave-l31f krb5kdc[3199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.67@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:41:38.141762 3703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:38.121630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:32783 (local address 127.0.197.126:40831)
0929 16:41:38.122165 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:38.122169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:38.123428 (+ 1259us) server_negotiation.cc:407] Connection header received
0929 16:41:38.125341 (+ 1913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:38.125346 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:38.125475 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:38.125601 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:38.127517 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:38.128297 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:38.130294 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:38.130964 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:38.131154 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:38.135070 (+ 3916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:38.135077 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:38.135084 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:38.135134 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:38.138131 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:38.139168 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:38.139176 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:38.139181 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:38.139331 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:38.139823 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:38.139828 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:38.139831 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:38.140127 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:38.140301 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:38.141093 (+ 792us) server_negotiation.cc:299] Negotiation successful
0929 16:41:38.141348 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":247,"threads_started":1}
I20260929 16:41:38.142792 3704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:38.122335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:40831 (local address 127.0.197.67:32783)
0929 16:41:38.123256 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:38.123298 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:38.125027 (+ 1729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:38.125876 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:38.125886 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:38.126456 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:41:38.127274 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:38.127301 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:38.128502 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:38.128506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:38.130117 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:38.130137 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:38.131280 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:38.131288 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:38.131570 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:38.132359 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:41:38.132400 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:41:38.134762 (+ 2362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:41:38.138450 (+ 3688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:38.138459 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:38.138477 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:38.138967 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:38.139471 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:38.139478 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:38.139486 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:38.139693 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:38.140272 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:41:38.140281 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:41:38.140703 (+ 422us) client_negotiation.cc:769] Sending connection context
0929 16:41:38.141007 (+ 304us) client_negotiation.cc:240] Negotiation successful
0929 16:41:38.141607 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":319,"threads_started":1}
I20260929 16:41:38.144682 3701 heartbeater.cc:347] Connected to a master server at 127.0.197.126:40831
I20260929 16:41:38.145045 3701 heartbeater.cc:464] Registering TS with master...
I20260929 16:41:38.145890 3701 heartbeater.cc:511] Master 127.0.197.126:40831 requested a full tablet report, sending...
I20260929 16:41:38.147730 3243 ts_manager.cc:194] Registered new tserver with Master: 575e23ee64034d40b3e93312c1d713c0 (127.0.197.67:36421)
I20260929 16:41:38.148667 3243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.67@KRBTEST.COM'} at 127.0.197.67:32783
I20260929 16:41:38.158352 789 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 16:41:38.192029 3703 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:41:38.189074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52926 (local address 127.0.197.126:40831)
0929 16:41:38.189456 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:38.189465 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:38.189830 (+ 365us) server_negotiation.cc:407] Connection header received
0929 16:41:38.190081 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:38.190085 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:38.190152 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:38.190271 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:38.191742 (+ 1471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52926: BlockingRecv error: recv got EOF from 127.0.0.1:52926 (error 108)
Metrics: {"server-negotiator.queue_time_us":209}
I20260929 16:41:38.195689 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 3305
W20260929 16:41:38.199184 789 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 3305: Not authorized: Client connection negotiation failed: client connection to 127.0.197.65:38931: server requires authentication, but client does not have Kerberos credentials available
I20260929 16:41:38.199426 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 3305
I20260929 16:41:38.215468 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 3439
W20260929 16:41:38.218446 789 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 3439: Not authorized: Client connection negotiation failed: client connection to 127.0.197.66:44557: server requires authentication, but client does not have Kerberos credentials available
I20260929 16:41:38.218626 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 3439
I20260929 16:41:38.234622 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 3573
W20260929 16:41:38.238157 789 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 3573: Not authorized: Client connection negotiation failed: client connection to 127.0.197.67:36421: server requires authentication, but client does not have Kerberos credentials available
I20260929 16:41:38.238330 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 3573
I20260929 16:41:38.254307 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 3212
I20260929 16:41:38.256165 3274 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:41:38.361618 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 3212
2026-09-29T16:41:38Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3865 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790700068255611-789-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:41:38 dist-test-slave-l31f krb5kdc[3719](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:41:38 dist-test-slave-l31f krb5kdc[3719](info): set up 2 sockets
Sep 29 16:41:38 dist-test-slave-l31f krb5kdc[3719](info): commencing operation
krb5kdc: starting...
W20260929 16:41:39.471457 789 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790700068255611-789-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790700068255611-789-0/krb5kdc/test-user.keytab.
Sep 29 16:41:39 dist-test-slave-l31f krb5kdc[3719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:41:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:41:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.197.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
I20260929 16:41:39.688434 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:34369
--webserver_interface=127.0.197.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:35997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.126:34369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:41:40.040609 3732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:41:40.040994 3732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:41:40.041091 3732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:41:40.048652 3732 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:41:40.048789 3732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:41:40.048915 3732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:41:40.049015 3732 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:41:40.049098 3732 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:41:40.060295 3732 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:35997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790700068255611-789-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.197.126:34369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:34369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.197.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:41:40.061791 3732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:41:40.063277 3732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:41:40.076635 3740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:40.076679 3738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:40.076679 3737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:41:40.077247 3732 server_base.cc:1034] running on GCE node
I20260929 16:41:40.078261 3732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:41:40.079944 3732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:41:40.081193 3732 hybrid_clock.cc:648] HybridClock initialized: now 1790700100081129 us; error 65 us; skew 500 ppm
I20260929 16:41:40.081636 3732 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:41:40 dist-test-slave-l31f krb5kdc[3719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700100, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:41:40.085124 3732 init.cc:377] Logged in from keytab as kudu/127.0.197.126@KRBTEST.COM (short username kudu)
I20260929 16:41:40.086934 3732 webserver.cc:460] Webserver started at http://127.0.197.126:39955/ using document root <none> and password file <none>
I20260929 16:41:40.087749 3732 fs_manager.cc:359] Metadata directory not provided
I20260929 16:41:40.087908 3732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:41:40.088227 3732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:41:40.091645 3732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790700068255611-789-0/minicluster-data/master-0/data/instance:
uuid: "ac01fab764894ec6a61fb9aafb8dcb67"
format_stamp: "Formatted at 2026-09-29 16:41:40 on dist-test-slave-l31f"
server_key: "d0117c15919f43ab40f7471a54a305fb"
server_key_iv: "e52bffc8f7add3930b4e321ce9ab60aa"
server_key_version: "encryptionkey@0"
I20260929 16:41:40.092593 3732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790700068255611-789-0/minicluster-data/master-0/wal/instance:
uuid: "ac01fab764894ec6a61fb9aafb8dcb67"
format_stamp: "Formatted at 2026-09-29 16:41:40 on dist-test-slave-l31f"
server_key: "d0117c15919f43ab40f7471a54a305fb"
server_key_iv: "e52bffc8f7add3930b4e321ce9ab60aa"
server_key_version: "encryptionkey@0"
I20260929 16:41:40.098579 3732 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 16:41:40.102808 3747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:40.104547 3732 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 16:41:40.104775 3732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "ac01fab764894ec6a61fb9aafb8dcb67"
format_stamp: "Formatted at 2026-09-29 16:41:40 on dist-test-slave-l31f"
server_key: "d0117c15919f43ab40f7471a54a305fb"
server_key_iv: "e52bffc8f7add3930b4e321ce9ab60aa"
server_key_version: "encryptionkey@0"
I20260929 16:41:40.105044 3732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:41:40.123224 3732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:41:40.128126 3732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:41:40.128454 3732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:41:40.148820 3732 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.126:34369
I20260929 16:41:40.148852 3798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.126:34369 every 8 connection(s)
I20260929 16:41:40.150348 3732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:41:40.154381 3799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:41:40.160458 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 3732
I20260929 16:41:40.160684 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790700068255611-789-0/minicluster-data/master-0/wal/instance
I20260929 16:41:40.161482 789 external_mini_cluster.cc:1428] Setting key fa3b563fbbb569816add6d307e892fd1
I20260929 16:41:40.168488 3799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac01fab764894ec6a61fb9aafb8dcb67: Bootstrap starting.
I20260929 16:41:40.172839 3799 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac01fab764894ec6a61fb9aafb8dcb67: Neither blocks nor log segments found. Creating new log.
I20260929 16:41:40.174273 3799 log.cc:824] T 00000000000000000000000000000000 P ac01fab764894ec6a61fb9aafb8dcb67: Log is configured to *not* fsync() on all Append() calls
I20260929 16:41:40.178162 3799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac01fab764894ec6a61fb9aafb8dcb67: No bootstrap required, opened a new log
Sep 29 16:41:40 dist-test-slave-l31f krb5kdc[3719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:41:40.187853 3799 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac01fab764894ec6a61fb9aafb8dcb67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac01fab764894ec6a61fb9aafb8dcb67" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 34369 } }
I20260929 16:41:40.188300 3799 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac01fab764894ec6a61fb9aafb8dcb67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:41:40.188417 3799 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac01fab764894ec6a61fb9aafb8dcb67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac01fab764894ec6a61fb9aafb8dcb67, State: Initialized, Role: FOLLOWER
I20260929 16:41:40.188563 3802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:40.163852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41484 (local address 127.0.197.126:34369)
0929 16:41:40.164797 (+ 945us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:40.164811 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:40.164856 (+ 45us) server_negotiation.cc:407] Connection header received
0929 16:41:40.166733 (+ 1877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:40.166760 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:40.167415 (+ 655us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:40.167845 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:40.169861 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:40.171469 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:40.173382 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:40.174173 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:40.174331 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:40.179701 (+ 5370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:40.179720 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:40.179740 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:40.179789 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:40.183363 (+ 3574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:40.184382 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:40.184390 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:40.184398 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:40.184605 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:40.185095 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:40.185100 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:40.185102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:40.185818 (+ 716us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:40.186125 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:40.186695 (+ 570us) server_negotiation.cc:299] Negotiation successful
0929 16:41:40.187250 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":475,"threads_started":1}
I20260929 16:41:40.189162 3799 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac01fab764894ec6a61fb9aafb8dcb67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac01fab764894ec6a61fb9aafb8dcb67" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 34369 } }
I20260929 16:41:40.189451 3799 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac01fab764894ec6a61fb9aafb8dcb67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:41:40.189610 3799 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac01fab764894ec6a61fb9aafb8dcb67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:41:40.189814 3799 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac01fab764894ec6a61fb9aafb8dcb67 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:41:40.193260 3799 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac01fab764894ec6a61fb9aafb8dcb67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac01fab764894ec6a61fb9aafb8dcb67" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 34369 } }
I20260929 16:41:40.193657 789 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:41:40.193980 3799 leader_election.cc:302] T 00000000000000000000000000000000 P ac01fab764894ec6a61fb9aafb8dcb67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac01fab764894ec6a61fb9aafb8dcb67; no voters:
I20260929 16:41:40.194701 3799 leader_election.cc:288] T 00000000000000000000000000000000 P ac01fab764894ec6a61fb9aafb8dcb67 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:41:40.194819 3804 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac01fab764894ec6a61fb9aafb8dcb67 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:41:40.195582 3804 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac01fab764894ec6a61fb9aafb8dcb67 [term 1 LEADER]: Becoming Leader. State: Replica: ac01fab764894ec6a61fb9aafb8dcb67, State: Running, Role: LEADER
I20260929 16:41:40.196280 3804 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac01fab764894ec6a61fb9aafb8dcb67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac01fab764894ec6a61fb9aafb8dcb67" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 34369 } }
I20260929 16:41:40.198230 3804 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac01fab764894ec6a61fb9aafb8dcb67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac01fab764894ec6a61fb9aafb8dcb67. Latest consensus state: current_term: 1 leader_uuid: "ac01fab764894ec6a61fb9aafb8dcb67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac01fab764894ec6a61fb9aafb8dcb67" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 34369 } } }
I20260929 16:41:40.198684 3804 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac01fab764894ec6a61fb9aafb8dcb67 [sys.catalog]: This master's current role is: LEADER
I20260929 16:41:40.199143 3806 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac01fab764894ec6a61fb9aafb8dcb67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac01fab764894ec6a61fb9aafb8dcb67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac01fab764894ec6a61fb9aafb8dcb67" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 34369 } } }
I20260929 16:41:40.199558 3806 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac01fab764894ec6a61fb9aafb8dcb67 [sys.catalog]: This master's current role is: LEADER
I20260929 16:41:40.203603 3799 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac01fab764894ec6a61fb9aafb8dcb67 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:41:40.205356 3808 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:41:40.211499 3808 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:41:40.230567 3808 catalog_manager.cc:1347] Generated new cluster ID: 9f96b4da248a44ccb54847b5302a071a
I20260929 16:41:40.230850 3808 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:41:40.243971 3824 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 16:41:40.256652 3802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:40.248614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41496 (local address 127.0.197.126:34369)
0929 16:41:40.248857 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:40.248871 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:40.249485 (+ 614us) server_negotiation.cc:407] Connection header received
0929 16:41:40.249690 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:40.249696 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:40.249775 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:40.249889 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:40.250949 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:40.251759 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:40.253121 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:40.253617 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:40.253771 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:40.254982 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:40.254987 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:40.254991 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:41:40.255746 (+ 755us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:40.255823 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:41:40.255852 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:40.255987 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:40.256127 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 16:41:40.256242 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 16:41:40.274467 3808 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:41:40.276002 3808 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:41:40.285370 3808 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac01fab764894ec6a61fb9aafb8dcb67: Generated new TSK 0
I20260929 16:41:40.286383 3808 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:41:40.302867 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 3732
I20260929 16:41:40.304267 3794 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:41:40.400331 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 3732
2026-09-29T16:41:40Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2031 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:41:40 dist-test-slave-l31f krb5kdc[3833](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:41:40 dist-test-slave-l31f krb5kdc[3833](info): set up 2 sockets
Sep 29 16:41:40 dist-test-slave-l31f krb5kdc[3833](info): commencing operation
krb5kdc: starting...
W20260929 16:41:41.503578 789 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.009s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/krb5kdc/test-user.keytab.
Sep 29 16:41:41 dist-test-slave-l31f krb5kdc[3833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:41:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:41:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.197.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
I20260929 16:41:41.723587 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:36345
--webserver_interface=127.0.197.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:38993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.126:36345
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:41:42.087841 3846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:41:42.088194 3846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:41:42.088358 3846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:41:42.096228 3846 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:41:42.096387 3846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:41:42.096438 3846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:41:42.096542 3846 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:41:42.096621 3846 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:41:42.107549 3846 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:38993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.197.126:36345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:36345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.197.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:41:42.109067 3846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:41:42.110459 3846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:41:42.123243 3851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:42.123191 3854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:42.123148 3852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:41:42.124316 3846 server_base.cc:1034] running on GCE node
I20260929 16:41:42.125270 3846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:41:42.127161 3846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:41:42.128468 3846 hybrid_clock.cc:648] HybridClock initialized: now 1790700102128402 us; error 72 us; skew 500 ppm
I20260929 16:41:42.128921 3846 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:41:42 dist-test-slave-l31f krb5kdc[3833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:41:42.132658 3846 init.cc:377] Logged in from keytab as kudu/127.0.197.126@KRBTEST.COM (short username kudu)
I20260929 16:41:42.134604 3846 webserver.cc:460] Webserver started at http://127.0.197.126:37589/ using document root <none> and password file <none>
I20260929 16:41:42.135507 3846 fs_manager.cc:359] Metadata directory not provided
I20260929 16:41:42.135641 3846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:41:42.135939 3846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:41:42.139564 3846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/master-0/data/instance:
uuid: "913cdb6f9903440590438091eddc233d"
format_stamp: "Formatted at 2026-09-29 16:41:42 on dist-test-slave-l31f"
server_key: "a627e02ce675d103d5fe3c3edc3a2b1e"
server_key_iv: "f16652f9298e691dc3441d6b4abc17e3"
server_key_version: "encryptionkey@0"
I20260929 16:41:42.140456 3846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/master-0/wal/instance:
uuid: "913cdb6f9903440590438091eddc233d"
format_stamp: "Formatted at 2026-09-29 16:41:42 on dist-test-slave-l31f"
server_key: "a627e02ce675d103d5fe3c3edc3a2b1e"
server_key_iv: "f16652f9298e691dc3441d6b4abc17e3"
server_key_version: "encryptionkey@0"
I20260929 16:41:42.146674 3846 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 16:41:42.150714 3861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:42.152416 3846 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:41:42.152653 3846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "913cdb6f9903440590438091eddc233d"
format_stamp: "Formatted at 2026-09-29 16:41:42 on dist-test-slave-l31f"
server_key: "a627e02ce675d103d5fe3c3edc3a2b1e"
server_key_iv: "f16652f9298e691dc3441d6b4abc17e3"
server_key_version: "encryptionkey@0"
I20260929 16:41:42.152863 3846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:41:42.169871 3846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:41:42.174963 3846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:41:42.175325 3846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:41:42.196758 3846 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.126:36345
I20260929 16:41:42.196810 3912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.126:36345 every 8 connection(s)
I20260929 16:41:42.198312 3846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:41:42.202535 3913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:41:42.205859 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 3846
I20260929 16:41:42.206091 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/master-0/wal/instance
I20260929 16:41:42.206899 789 external_mini_cluster.cc:1428] Setting key 8c0dca06cc5ffb29ffd41614f6100134
I20260929 16:41:42.216161 3913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 913cdb6f9903440590438091eddc233d: Bootstrap starting.
I20260929 16:41:42.219782 3913 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 913cdb6f9903440590438091eddc233d: Neither blocks nor log segments found. Creating new log.
I20260929 16:41:42.221112 3913 log.cc:824] T 00000000000000000000000000000000 P 913cdb6f9903440590438091eddc233d: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:41:42 dist-test-slave-l31f krb5kdc[3833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:41:42.224790 3913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 913cdb6f9903440590438091eddc233d: No bootstrap required, opened a new log
I20260929 16:41:42.232642 3916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:42.209411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34194 (local address 127.0.197.126:36345)
0929 16:41:42.210272 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:42.210285 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:42.210330 (+ 45us) server_negotiation.cc:407] Connection header received
0929 16:41:42.212147 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:42.212173 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:42.212650 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:42.213104 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:42.214618 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:42.216787 (+ 2169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:42.218428 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:42.219305 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:42.219529 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:42.224344 (+ 4815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:42.224352 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:42.224369 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:42.224410 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:42.228011 (+ 3601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:42.228894 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:42.228901 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:42.228909 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:42.229093 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:42.229491 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:42.229501 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:42.229503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:42.230135 (+ 632us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:42.230491 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:42.230993 (+ 502us) server_negotiation.cc:299] Negotiation successful
0929 16:41:42.231487 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":384,"threads_started":1}
I20260929 16:41:42.234107 3913 raft_consensus.cc:348] T 00000000000000000000000000000000 P 913cdb6f9903440590438091eddc233d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "913cdb6f9903440590438091eddc233d" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 36345 } }
I20260929 16:41:42.234534 3913 raft_consensus.cc:374] T 00000000000000000000000000000000 P 913cdb6f9903440590438091eddc233d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:41:42.234673 3913 raft_consensus.cc:729] T 00000000000000000000000000000000 P 913cdb6f9903440590438091eddc233d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 913cdb6f9903440590438091eddc233d, State: Initialized, Role: FOLLOWER
I20260929 16:41:42.235399 3913 consensus_queue.cc:260] T 00000000000000000000000000000000 P 913cdb6f9903440590438091eddc233d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "913cdb6f9903440590438091eddc233d" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 36345 } }
I20260929 16:41:42.235765 3913 raft_consensus.cc:388] T 00000000000000000000000000000000 P 913cdb6f9903440590438091eddc233d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:41:42.235934 3913 raft_consensus.cc:482] T 00000000000000000000000000000000 P 913cdb6f9903440590438091eddc233d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:41:42.236138 3913 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 913cdb6f9903440590438091eddc233d [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:41:42.238716 3913 raft_consensus.cc:504] T 00000000000000000000000000000000 P 913cdb6f9903440590438091eddc233d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "913cdb6f9903440590438091eddc233d" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 36345 } }
I20260929 16:41:42.239419 3913 leader_election.cc:302] T 00000000000000000000000000000000 P 913cdb6f9903440590438091eddc233d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 913cdb6f9903440590438091eddc233d; no voters:
I20260929 16:41:42.240100 3913 leader_election.cc:288] T 00000000000000000000000000000000 P 913cdb6f9903440590438091eddc233d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:41:42.240191 3918 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 913cdb6f9903440590438091eddc233d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:41:42.241019 3918 raft_consensus.cc:686] T 00000000000000000000000000000000 P 913cdb6f9903440590438091eddc233d [term 1 LEADER]: Becoming Leader. State: Replica: 913cdb6f9903440590438091eddc233d, State: Running, Role: LEADER
I20260929 16:41:42.241591 3918 consensus_queue.cc:237] T 00000000000000000000000000000000 P 913cdb6f9903440590438091eddc233d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "913cdb6f9903440590438091eddc233d" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 36345 } }
I20260929 16:41:42.242497 3913 sys_catalog.cc:564] T 00000000000000000000000000000000 P 913cdb6f9903440590438091eddc233d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:41:42.244975 3919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 913cdb6f9903440590438091eddc233d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "913cdb6f9903440590438091eddc233d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "913cdb6f9903440590438091eddc233d" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 36345 } } }
I20260929 16:41:42.245069 3920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 913cdb6f9903440590438091eddc233d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 913cdb6f9903440590438091eddc233d. Latest consensus state: current_term: 1 leader_uuid: "913cdb6f9903440590438091eddc233d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "913cdb6f9903440590438091eddc233d" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 36345 } } }
I20260929 16:41:42.245682 3920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 913cdb6f9903440590438091eddc233d [sys.catalog]: This master's current role is: LEADER
I20260929 16:41:42.246688 3919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 913cdb6f9903440590438091eddc233d [sys.catalog]: This master's current role is: LEADER
I20260929 16:41:42.251839 3924 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:41:42.257246 3924 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:41:42.267208 3924 catalog_manager.cc:1347] Generated new cluster ID: 4c5c926b62904ae3995e699b3ffee5f6
I20260929 16:41:42.267364 3924 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:41:42.298166 3924 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:41:42.299724 3924 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:41:42.309166 3924 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 913cdb6f9903440590438091eddc233d: Generated new TSK 0
I20260929 16:41:42.310220 3924 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.197.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab.
Entry for principal kudu/127.0.197.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab.
I20260929 16:41:42.375953 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.65:0
--local_ip_for_outbound_sockets=127.0.197.65
--webserver_interface=127.0.197.65
--webserver_port=0
--tserver_master_addrs=127.0.197.126:36345
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab
--principal=kudu/127.0.197.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:38993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:41:42.737371 3939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:41:42.737697 3939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:41:42.737797 3939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:41:42.745409 3939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:41:42.745548 3939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:41:42.745667 3939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.65
I20260929 16:41:42.756747 3939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:38993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab
--principal=kudu/127.0.197.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.197.65
--webserver_port=0
--tserver_master_addrs=127.0.197.126:36345
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.65
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:41:42.758157 3939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:41:42.759624 3939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:41:42.773598 3944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:42.773492 3947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:42.774148 3945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:41:42.775625 3939 server_base.cc:1034] running on GCE node
I20260929 16:41:42.776371 3939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:41:42.777477 3939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:41:42.778740 3939 hybrid_clock.cc:648] HybridClock initialized: now 1790700102778693 us; error 44 us; skew 500 ppm
I20260929 16:41:42.779273 3939 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:41:42 dist-test-slave-l31f krb5kdc[3833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:41:42.789088 3939 init.cc:377] Logged in from keytab as kudu/127.0.197.65@KRBTEST.COM (short username kudu)
I20260929 16:41:42.790946 3939 webserver.cc:460] Webserver started at http://127.0.197.65:38575/ using document root <none> and password file <none>
I20260929 16:41:42.792101 3939 fs_manager.cc:359] Metadata directory not provided
I20260929 16:41:42.792276 3939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:41:42.792605 3939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:41:42.795884 3939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-0/data/instance:
uuid: "caa00f55da324759a4ac29baeb15e984"
format_stamp: "Formatted at 2026-09-29 16:41:42 on dist-test-slave-l31f"
server_key: "1fe826cff40cb2d8f147b24448b62e5d"
server_key_iv: "082815dedac9897176948f64229a2078"
server_key_version: "encryptionkey@0"
I20260929 16:41:42.796795 3939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-0/wal/instance:
uuid: "caa00f55da324759a4ac29baeb15e984"
format_stamp: "Formatted at 2026-09-29 16:41:42 on dist-test-slave-l31f"
server_key: "1fe826cff40cb2d8f147b24448b62e5d"
server_key_iv: "082815dedac9897176948f64229a2078"
server_key_version: "encryptionkey@0"
I20260929 16:41:42.802481 3939 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 16:41:42.806720 3954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:42.808946 3939 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:41:42.809142 3939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-0/wal
uuid: "caa00f55da324759a4ac29baeb15e984"
format_stamp: "Formatted at 2026-09-29 16:41:42 on dist-test-slave-l31f"
server_key: "1fe826cff40cb2d8f147b24448b62e5d"
server_key_iv: "082815dedac9897176948f64229a2078"
server_key_version: "encryptionkey@0"
I20260929 16:41:42.809397 3939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:41:42.825228 3939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:41:42.830812 3939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:41:42.831315 3939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:41:42.832384 3939 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:41:42.833933 3939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:41:42.834016 3939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:42.834111 3939 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:41:42.834163 3939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:42.873350 3939 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.65:37899
I20260929 16:41:42.873400 4066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.65:37899 every 8 connection(s)
I20260929 16:41:42.874768 3939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
I20260929 16:41:42.878577 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 3939
I20260929 16:41:42.878803 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-0/wal/instance
I20260929 16:41:42.880091 789 external_mini_cluster.cc:1428] Setting key 35c20ce5de2698f2db6d986e629c0477
Sep 29 16:41:42 dist-test-slave-l31f krb5kdc[3833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.65@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:41:42.901543 4069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:42.877958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:59403 (local address 127.0.197.126:36345)
0929 16:41:42.878470 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:42.878477 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:42.880474 (+ 1997us) server_negotiation.cc:407] Connection header received
0929 16:41:42.883159 (+ 2685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:42.883163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:42.883236 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:42.883340 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:42.885896 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:42.886663 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:42.889792 (+ 3129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:42.890252 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:42.890415 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:42.894627 (+ 4212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:42.894633 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:42.894637 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:42.894678 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:42.897923 (+ 3245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:42.898890 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:42.898895 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:42.898898 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:42.899118 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:42.899613 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:42.899619 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:42.899623 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:42.899915 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:42.900089 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:42.900886 (+ 797us) server_negotiation.cc:299] Negotiation successful
0929 16:41:42.901158 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":270,"threads_started":1}
I20260929 16:41:42.902658 4070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:42.878622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36345 (local address 127.0.197.65:59403)
0929 16:41:42.880163 (+ 1541us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:42.880259 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:42.882721 (+ 2462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:42.883686 (+ 965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:42.883704 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:42.884634 (+ 930us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:41:42.885637 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:42.885672 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:42.886863 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:42.886872 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:42.889563 (+ 2691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:42.889582 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:42.890562 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:42.890571 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:42.890839 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:42.891699 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:41:42.891729 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:41:42.894304 (+ 2575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:41:42.898184 (+ 3880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:42.898194 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:42.898212 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:42.898726 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:42.899244 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:42.899250 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:42.899254 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:42.899480 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:42.900081 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:41:42.900091 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:41:42.900483 (+ 392us) client_negotiation.cc:769] Sending connection context
0929 16:41:42.900765 (+ 282us) client_negotiation.cc:240] Negotiation successful
0929 16:41:42.901381 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1021,"thread_start_us":654,"threads_started":1}
WARNING: no policy specified for kudu/127.0.197.66@KRBTEST.COM; defaulting to no policy
I20260929 16:41:42.904876 4067 heartbeater.cc:347] Connected to a master server at 127.0.197.126:36345
I20260929 16:41:42.905251 4067 heartbeater.cc:464] Registering TS with master...
I20260929 16:41:42.906101 4067 heartbeater.cc:511] Master 127.0.197.126:36345 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.66@KRBTEST.COM" created.
I20260929 16:41:42.908842 3878 ts_manager.cc:194] Registered new tserver with Master: caa00f55da324759a4ac29baeb15e984 (127.0.197.65:37899)
I20260929 16:41:42.910521 3878 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.65@KRBTEST.COM'} at 127.0.197.65:59403
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab.
Entry for principal kudu/127.0.197.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab.
I20260929 16:41:42.935122 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.66:0
--local_ip_for_outbound_sockets=127.0.197.66
--webserver_interface=127.0.197.66
--webserver_port=0
--tserver_master_addrs=127.0.197.126:36345
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab
--principal=kudu/127.0.197.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:38993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:41:43.291270 4073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:41:43.291646 4073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:41:43.291808 4073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:41:43.299355 4073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:41:43.299559 4073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:41:43.299753 4073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.66
I20260929 16:41:43.309924 4073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:38993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab
--principal=kudu/127.0.197.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.197.66
--webserver_port=0
--tserver_master_addrs=127.0.197.126:36345
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.66
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:41:43.311465 4073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:41:43.312845 4073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:41:43.327220 4079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:43.327242 4078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:43.327163 4081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:41:43.328150 4073 server_base.cc:1034] running on GCE node
I20260929 16:41:43.328903 4073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:41:43.329977 4073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:41:43.331208 4073 hybrid_clock.cc:648] HybridClock initialized: now 1790700103331156 us; error 49 us; skew 500 ppm
I20260929 16:41:43.331559 4073 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:41:43 dist-test-slave-l31f krb5kdc[3833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:41:43.335309 4073 init.cc:377] Logged in from keytab as kudu/127.0.197.66@KRBTEST.COM (short username kudu)
I20260929 16:41:43.337019 4073 webserver.cc:460] Webserver started at http://127.0.197.66:41669/ using document root <none> and password file <none>
I20260929 16:41:43.337785 4073 fs_manager.cc:359] Metadata directory not provided
I20260929 16:41:43.337946 4073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:41:43.338263 4073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:41:43.341482 4073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-1/data/instance:
uuid: "a48edf32afb443a0904e9a350a71fb2d"
format_stamp: "Formatted at 2026-09-29 16:41:43 on dist-test-slave-l31f"
server_key: "82f584045a543f32381c9ca240900063"
server_key_iv: "c39d15aa62faabb6479421896a090427"
server_key_version: "encryptionkey@0"
I20260929 16:41:43.342394 4073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-1/wal/instance:
uuid: "a48edf32afb443a0904e9a350a71fb2d"
format_stamp: "Formatted at 2026-09-29 16:41:43 on dist-test-slave-l31f"
server_key: "82f584045a543f32381c9ca240900063"
server_key_iv: "c39d15aa62faabb6479421896a090427"
server_key_version: "encryptionkey@0"
I20260929 16:41:43.348412 4073 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 16:41:43.352533 4088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:43.354199 4073 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:41:43.354441 4073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-1/wal
uuid: "a48edf32afb443a0904e9a350a71fb2d"
format_stamp: "Formatted at 2026-09-29 16:41:43 on dist-test-slave-l31f"
server_key: "82f584045a543f32381c9ca240900063"
server_key_iv: "c39d15aa62faabb6479421896a090427"
server_key_version: "encryptionkey@0"
I20260929 16:41:43.354672 4073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:41:43.381395 4073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:41:43.386296 4073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:41:43.386628 4073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:41:43.387725 4073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:41:43.389132 4073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:41:43.389248 4073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:43.389395 4073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:41:43.389472 4073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:43.424010 4073 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.66:40307
I20260929 16:41:43.424144 4200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.66:40307 every 8 connection(s)
I20260929 16:41:43.425810 4073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
I20260929 16:41:43.426532 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 4073
I20260929 16:41:43.426704 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-1/wal/instance
I20260929 16:41:43.427520 789 external_mini_cluster.cc:1428] Setting key a8dfae2e707e15181236b6886aba2a49
Sep 29 16:41:43 dist-test-slave-l31f krb5kdc[3833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.66@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.197.67@KRBTEST.COM; defaulting to no policy
I20260929 16:41:43.448177 4203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:43.428923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:41349 (local address 127.0.197.126:36345)
0929 16:41:43.429390 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:43.429396 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:43.430735 (+ 1339us) server_negotiation.cc:407] Connection header received
0929 16:41:43.432781 (+ 2046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:43.432786 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:43.432877 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:43.433012 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:43.434675 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:43.435562 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:43.437207 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:43.437648 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:43.437786 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:43.441812 (+ 4026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:43.441817 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:43.441821 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:43.441856 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:43.444866 (+ 3010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:43.445723 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:43.445728 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:43.445731 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:43.445868 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:43.446362 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:43.446367 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:43.446369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:43.446662 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:43.446804 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:43.447655 (+ 851us) server_negotiation.cc:299] Negotiation successful
0929 16:41:43.447846 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":233,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.67@KRBTEST.COM" created.
I20260929 16:41:43.449425 4204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:43.429629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36345 (local address 127.0.197.66:41349)
0929 16:41:43.430574 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:43.430616 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:43.432459 (+ 1843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:43.433280 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:43.433290 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:43.433857 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:41:43.434479 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:43.434498 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:43.435736 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:43.435740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:43.437043 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:43.437056 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:43.437914 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:43.437922 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:43.438159 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:43.438923 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:41:43.438947 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:41:43.441530 (+ 2583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:41:43.445049 (+ 3519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:43.445060 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:43.445086 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:43.445573 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:43.446028 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:43.446032 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:43.446035 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:43.446237 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:43.446790 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:41:43.446796 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:41:43.447278 (+ 482us) client_negotiation.cc:769] Sending connection context
0929 16:41:43.447556 (+ 278us) client_negotiation.cc:240] Negotiation successful
0929 16:41:43.448021 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":399,"threads_started":1}
I20260929 16:41:43.451599 4201 heartbeater.cc:347] Connected to a master server at 127.0.197.126:36345
I20260929 16:41:43.452018 4201 heartbeater.cc:464] Registering TS with master...
I20260929 16:41:43.452994 4201 heartbeater.cc:511] Master 127.0.197.126:36345 requested a full tablet report, sending...
I20260929 16:41:43.454813 3878 ts_manager.cc:194] Registered new tserver with Master: a48edf32afb443a0904e9a350a71fb2d (127.0.197.66:40307)
I20260929 16:41:43.455875 3878 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.66@KRBTEST.COM'} at 127.0.197.66:41349
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab.
Entry for principal kudu/127.0.197.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab.
I20260929 16:41:43.479671 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.67:0
--local_ip_for_outbound_sockets=127.0.197.67
--webserver_interface=127.0.197.67
--webserver_port=0
--tserver_master_addrs=127.0.197.126:36345
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab
--principal=kudu/127.0.197.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:38993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:41:43.839476 4207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:41:43.839823 4207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:41:43.839990 4207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:41:43.847931 4207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:41:43.848117 4207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:41:43.848295 4207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.67
I20260929 16:41:43.860774 4207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:38993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab
--principal=kudu/127.0.197.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.197.67
--webserver_port=0
--tserver_master_addrs=127.0.197.126:36345
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.67
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:41:43.862263 4207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:41:43.863703 4207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:41:43.877450 4212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:43.877451 4213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:43.878382 4215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:41:43.879926 4207 server_base.cc:1034] running on GCE node
I20260929 16:41:43.880690 4207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:41:43.881948 4207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:41:43.883319 4207 hybrid_clock.cc:648] HybridClock initialized: now 1790700103883187 us; error 115 us; skew 500 ppm
I20260929 16:41:43.883705 4207 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:41:43 dist-test-slave-l31f krb5kdc[3833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:41:43.887171 4207 init.cc:377] Logged in from keytab as kudu/127.0.197.67@KRBTEST.COM (short username kudu)
I20260929 16:41:43.888851 4207 webserver.cc:460] Webserver started at http://127.0.197.67:35207/ using document root <none> and password file <none>
I20260929 16:41:43.889636 4207 fs_manager.cc:359] Metadata directory not provided
I20260929 16:41:43.889789 4207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:41:43.890120 4207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:41:43.893388 4207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-2/data/instance:
uuid: "8a90db0ced034203b4e8696094593e4d"
format_stamp: "Formatted at 2026-09-29 16:41:43 on dist-test-slave-l31f"
server_key: "626c2e9f7e1c1f943a570484f4ade380"
server_key_iv: "163168cceeba32bd860b33d54a9a5678"
server_key_version: "encryptionkey@0"
I20260929 16:41:43.894377 4207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-2/wal/instance:
uuid: "8a90db0ced034203b4e8696094593e4d"
format_stamp: "Formatted at 2026-09-29 16:41:43 on dist-test-slave-l31f"
server_key: "626c2e9f7e1c1f943a570484f4ade380"
server_key_iv: "163168cceeba32bd860b33d54a9a5678"
server_key_version: "encryptionkey@0"
I20260929 16:41:43.900766 4207 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260929 16:41:43.905092 4222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:43.906929 4207 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260929 16:41:43.907225 4207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-2/wal
uuid: "8a90db0ced034203b4e8696094593e4d"
format_stamp: "Formatted at 2026-09-29 16:41:43 on dist-test-slave-l31f"
server_key: "626c2e9f7e1c1f943a570484f4ade380"
server_key_iv: "163168cceeba32bd860b33d54a9a5678"
server_key_version: "encryptionkey@0"
I20260929 16:41:43.907449 4207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:41:43.914026 4067 heartbeater.cc:503] Master 127.0.197.126:36345 was elected leader, sending a full tablet report...
I20260929 16:41:43.927598 4207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:41:43.933917 4207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:41:43.934297 4207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:41:43.935727 4207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:41:43.937804 4207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:41:43.937918 4207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:43.938062 4207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:41:43.938134 4207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:43.988667 4207 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.67:40587
I20260929 16:41:43.988780 4334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.67:40587 every 8 connection(s)
I20260929 16:41:43.990051 4207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
I20260929 16:41:43.996224 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 4207
I20260929 16:41:43.996446 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790700068255611-789-0/minicluster-data/ts-2/wal/instance
I20260929 16:41:43.996891 789 external_mini_cluster.cc:1428] Setting key 484604b5543635be107d2eaede87c9aa
Sep 29 16:41:44 dist-test-slave-l31f krb5kdc[3833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.67@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:41:44.017417 4337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:43.993680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:37001 (local address 127.0.197.126:36345)
0929 16:41:43.994258 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:43.994267 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:43.995623 (+ 1356us) server_negotiation.cc:407] Connection header received
0929 16:41:43.998766 (+ 3143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:43.998772 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:43.998869 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:43.999063 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:44.001539 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:44.002302 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:44.005231 (+ 2929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:44.005665 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:44.005783 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:44.010413 (+ 4630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:44.010419 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:44.010422 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:44.010459 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:44.014023 (+ 3564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:44.014869 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:44.014874 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:44.014878 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:44.015058 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:44.015563 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:44.015568 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:44.015571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:44.015869 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:44.016026 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:44.016782 (+ 756us) server_negotiation.cc:299] Negotiation successful
0929 16:41:44.017011 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":256,"threads_started":1}
I20260929 16:41:44.018471 4338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:43.994354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36345 (local address 127.0.197.67:37001)
0929 16:41:43.995387 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:43.995451 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:43.998384 (+ 2933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:43.999428 (+ 1044us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:43.999446 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:44.000344 (+ 898us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:41:44.001314 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:44.001349 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:44.002493 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:44.002501 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:44.005006 (+ 2505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:44.005030 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:44.005952 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:44.005961 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:44.006333 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:44.007326 (+ 993us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:41:44.007361 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:41:44.010096 (+ 2735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:41:44.014190 (+ 4094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:44.014201 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:44.014222 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:44.014707 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:44.015196 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:44.015201 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:44.015205 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:44.015436 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:44.016008 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:41:44.016017 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:41:44.016407 (+ 390us) client_negotiation.cc:769] Sending connection context
0929 16:41:44.016689 (+ 282us) client_negotiation.cc:240] Negotiation successful
0929 16:41:44.017281 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":413,"threads_started":1}
I20260929 16:41:44.020130 4335 heartbeater.cc:347] Connected to a master server at 127.0.197.126:36345
I20260929 16:41:44.020474 4335 heartbeater.cc:464] Registering TS with master...
I20260929 16:41:44.021222 4335 heartbeater.cc:511] Master 127.0.197.126:36345 requested a full tablet report, sending...
I20260929 16:41:44.022724 3878 ts_manager.cc:194] Registered new tserver with Master: 8a90db0ced034203b4e8696094593e4d (127.0.197.67:40587)
I20260929 16:41:44.023712 3878 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.67@KRBTEST.COM'} at 127.0.197.67:37001
I20260929 16:41:44.025857 789 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 16:41:44 dist-test-slave-l31f krb5kdc[3833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700104, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 29 16:41:44 dist-test-slave-l31f krb5kdc[3833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700104, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:41:44.078320 4337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:44.062572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34196 (local address 127.0.197.126:36345)
0929 16:41:44.062787 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:44.062798 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:44.063279 (+ 481us) server_negotiation.cc:407] Connection header received
0929 16:41:44.063452 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:44.063457 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:44.063546 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:44.063651 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:44.065063 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:44.065826 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:44.067356 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:44.067838 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:44.067970 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:44.072684 (+ 4714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:44.072691 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:44.072695 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:44.072740 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:44.075790 (+ 3050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:44.076563 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:44.076567 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:44.076570 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:44.076705 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:44.077048 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:44.077051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:44.077053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:44.077400 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:44.077594 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:44.077860 (+ 266us) server_negotiation.cc:299] Negotiation successful
0929 16:41:44.078066 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260929 16:41:44.078936 3878 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:34196
I20260929 16:41:44.081167 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 3939
Sep 29 16:41:44 dist-test-slave-l31f krb5kdc[3833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700104, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.197.65@KRBTEST.COM
I20260929 16:41:44.100008 4347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:44.082834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43128 (local address 127.0.197.65:37899)
0929 16:41:44.083449 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:44.083456 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:44.083484 (+ 28us) server_negotiation.cc:407] Connection header received
0929 16:41:44.083545 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:44.083552 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:44.083777 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:44.083926 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:44.085291 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:44.086370 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:44.087880 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:44.088545 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:44.088659 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:44.093372 (+ 4713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:44.093389 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:44.093406 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:44.093457 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:44.096807 (+ 3350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:44.097653 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:44.097661 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:44.097664 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:44.097810 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:44.098216 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:44.098219 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:44.098222 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:44.098620 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:44.098857 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:44.099312 (+ 455us) server_negotiation.cc:299] Negotiation successful
0929 16:41:44.099598 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":327,"threads_started":1}
W20260929 16:41:44.101406 4062 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43128
W20260929 16:41:44.102701 789 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 3939: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:41:44.102967 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 3939
I20260929 16:41:44.119378 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 4073
Sep 29 16:41:44 dist-test-slave-l31f krb5kdc[3833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700104, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.197.66@KRBTEST.COM
I20260929 16:41:44.138343 4348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:44.120328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34014 (local address 127.0.197.66:40307)
0929 16:41:44.120971 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:44.120979 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:44.121006 (+ 27us) server_negotiation.cc:407] Connection header received
0929 16:41:44.121109 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:44.121122 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:44.121479 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:44.121635 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:44.123281 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:44.124561 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:44.126136 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:44.126786 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:44.126926 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:44.131624 (+ 4698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:44.131640 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:44.131659 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:44.131713 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:44.135331 (+ 3618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:44.136227 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:44.136235 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:44.136238 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:44.136377 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:44.136770 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:44.136774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:44.136775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:44.137219 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:44.137491 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:44.137760 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 16:41:44.138048 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":328,"threads_started":1}
W20260929 16:41:44.139788 4196 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:34014
W20260929 16:41:44.141088 789 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 4073: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:41:44.141250 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 4073
I20260929 16:41:44.158078 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 4207
Sep 29 16:41:44 dist-test-slave-l31f krb5kdc[3833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700104, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.197.67@KRBTEST.COM
I20260929 16:41:44.177765 4349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:44.159885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46482 (local address 127.0.197.67:40587)
0929 16:41:44.160598 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:44.160606 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:44.160640 (+ 34us) server_negotiation.cc:407] Connection header received
0929 16:41:44.160711 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:44.160718 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:44.160996 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:44.161174 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:44.162523 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:44.163924 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:44.165437 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:44.166111 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:44.166233 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:44.171270 (+ 5037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:44.171289 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:44.171307 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:44.171354 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:44.174639 (+ 3285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:44.175592 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:44.175601 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:44.175604 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:44.175749 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:44.176115 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:44.176119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:44.176121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:44.176524 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:44.176773 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:44.177209 (+ 436us) server_negotiation.cc:299] Negotiation successful
0929 16:41:44.177442 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":382,"threads_started":1}
W20260929 16:41:44.179702 4330 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:46482
W20260929 16:41:44.181452 789 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 4207: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:41:44.181680 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 4207
I20260929 16:41:44.198233 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 3846
I20260929 16:41:44.199640 3908 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:41:44.303496 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 3846
2026-09-29T16:41:44Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3912 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:41:44 dist-test-slave-l31f krb5kdc[4353](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:41:44 dist-test-slave-l31f krb5kdc[4353](info): set up 2 sockets
Sep 29 16:41:44 dist-test-slave-l31f krb5kdc[4353](info): commencing operation
krb5kdc: starting...
W20260929 16:41:45.441356 789 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/krb5kdc/test-user.keytab.
Sep 29 16:41:45 dist-test-slave-l31f krb5kdc[4353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:41:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:41:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.197.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
I20260929 16:41:45.680469 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:38675
--webserver_interface=127.0.197.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:35741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.126:38675
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:41:46.038666 4366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:41:46.039392 4366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:41:46.039580 4366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:41:46.049388 4366 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:41:46.049594 4366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:41:46.049710 4366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:41:46.049809 4366 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:41:46.049901 4366 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:41:46.062983 4366 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:35741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.197.126:38675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:38675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.197.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:41:46.064597 4366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:41:46.066002 4366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:41:46.080204 4371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:46.080252 4372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:46.080266 4374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:41:46.081120 4366 server_base.cc:1034] running on GCE node
I20260929 16:41:46.082062 4366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:41:46.083860 4366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:41:46.085124 4366 hybrid_clock.cc:648] HybridClock initialized: now 1790700106085045 us; error 67 us; skew 500 ppm
I20260929 16:41:46.085582 4366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:41:46 dist-test-slave-l31f krb5kdc[4353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700106, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:41:46.089166 4366 init.cc:377] Logged in from keytab as kudu/127.0.197.126@KRBTEST.COM (short username kudu)
I20260929 16:41:46.090952 4366 webserver.cc:460] Webserver started at http://127.0.197.126:35225/ using document root <none> and password file <none>
I20260929 16:41:46.091796 4366 fs_manager.cc:359] Metadata directory not provided
I20260929 16:41:46.091919 4366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:41:46.092200 4366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:41:46.095572 4366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/master-0/data/instance:
uuid: "0b2ccbf9f4214610b14f31573df796b4"
format_stamp: "Formatted at 2026-09-29 16:41:46 on dist-test-slave-l31f"
server_key: "5b4a012754d3c5a36e1128b6b56dddcf"
server_key_iv: "eda7c74faea9543472ca3362020cfafc"
server_key_version: "encryptionkey@0"
I20260929 16:41:46.096405 4366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/master-0/wal/instance:
uuid: "0b2ccbf9f4214610b14f31573df796b4"
format_stamp: "Formatted at 2026-09-29 16:41:46 on dist-test-slave-l31f"
server_key: "5b4a012754d3c5a36e1128b6b56dddcf"
server_key_iv: "eda7c74faea9543472ca3362020cfafc"
server_key_version: "encryptionkey@0"
I20260929 16:41:46.102682 4366 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 16:41:46.106854 4381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:46.108664 4366 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260929 16:41:46.108897 4366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "0b2ccbf9f4214610b14f31573df796b4"
format_stamp: "Formatted at 2026-09-29 16:41:46 on dist-test-slave-l31f"
server_key: "5b4a012754d3c5a36e1128b6b56dddcf"
server_key_iv: "eda7c74faea9543472ca3362020cfafc"
server_key_version: "encryptionkey@0"
I20260929 16:41:46.109125 4366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:41:46.142297 4366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:41:46.147586 4366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:41:46.147926 4366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:41:46.171123 4366 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.126:38675
I20260929 16:41:46.171188 4432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.126:38675 every 8 connection(s)
I20260929 16:41:46.172775 4366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:41:46.173190 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 4366
I20260929 16:41:46.173452 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/master-0/wal/instance
I20260929 16:41:46.174398 789 external_mini_cluster.cc:1428] Setting key 71602b0d7ef9ef89443b029c9f47f7e5
I20260929 16:41:46.178864 4433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 16:41:46 dist-test-slave-l31f krb5kdc[4353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:41:46.191514 4433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b2ccbf9f4214610b14f31573df796b4: Bootstrap starting.
I20260929 16:41:46.195214 4433 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b2ccbf9f4214610b14f31573df796b4: Neither blocks nor log segments found. Creating new log.
I20260929 16:41:46.196354 4433 log.cc:824] T 00000000000000000000000000000000 P 0b2ccbf9f4214610b14f31573df796b4: Log is configured to *not* fsync() on all Append() calls
I20260929 16:41:46.199769 4433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b2ccbf9f4214610b14f31573df796b4: No bootstrap required, opened a new log
I20260929 16:41:46.200045 4436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:46.177151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34804 (local address 127.0.197.126:38675)
0929 16:41:46.178111 (+ 960us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:46.178126 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:46.178173 (+ 47us) server_negotiation.cc:407] Connection header received
0929 16:41:46.180070 (+ 1897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:46.180097 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:46.180550 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:46.181006 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:46.182596 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:46.184167 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:46.185514 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:46.186558 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:46.186877 (+ 319us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:46.191851 (+ 4974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:46.191860 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:46.191879 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:46.191937 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:46.195373 (+ 3436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:46.196216 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:46.196223 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:46.196230 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:46.196431 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:46.196843 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:46.196849 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:46.196852 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:46.197447 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:46.197804 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:46.198317 (+ 513us) server_negotiation.cc:299] Negotiation successful
0929 16:41:46.198760 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":721,"thread_start_us":445,"threads_started":1}
I20260929 16:41:46.210201 4433 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b2ccbf9f4214610b14f31573df796b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b2ccbf9f4214610b14f31573df796b4" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 38675 } }
I20260929 16:41:46.210707 4433 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b2ccbf9f4214610b14f31573df796b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:41:46.210842 4433 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b2ccbf9f4214610b14f31573df796b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b2ccbf9f4214610b14f31573df796b4, State: Initialized, Role: FOLLOWER
I20260929 16:41:46.211597 4433 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b2ccbf9f4214610b14f31573df796b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b2ccbf9f4214610b14f31573df796b4" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 38675 } }
I20260929 16:41:46.211903 4433 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b2ccbf9f4214610b14f31573df796b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:41:46.212074 4433 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b2ccbf9f4214610b14f31573df796b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:41:46.212273 4433 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b2ccbf9f4214610b14f31573df796b4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:41:46.215312 4433 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b2ccbf9f4214610b14f31573df796b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b2ccbf9f4214610b14f31573df796b4" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 38675 } }
I20260929 16:41:46.215962 4433 leader_election.cc:302] T 00000000000000000000000000000000 P 0b2ccbf9f4214610b14f31573df796b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b2ccbf9f4214610b14f31573df796b4; no voters:
I20260929 16:41:46.216651 4433 leader_election.cc:288] T 00000000000000000000000000000000 P 0b2ccbf9f4214610b14f31573df796b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:41:46.216812 4438 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b2ccbf9f4214610b14f31573df796b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:41:46.217548 4438 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b2ccbf9f4214610b14f31573df796b4 [term 1 LEADER]: Becoming Leader. State: Replica: 0b2ccbf9f4214610b14f31573df796b4, State: Running, Role: LEADER
I20260929 16:41:46.218151 4438 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b2ccbf9f4214610b14f31573df796b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b2ccbf9f4214610b14f31573df796b4" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 38675 } }
I20260929 16:41:46.219085 4433 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b2ccbf9f4214610b14f31573df796b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:41:46.221560 4439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b2ccbf9f4214610b14f31573df796b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b2ccbf9f4214610b14f31573df796b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b2ccbf9f4214610b14f31573df796b4" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 38675 } } }
I20260929 16:41:46.221556 4440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b2ccbf9f4214610b14f31573df796b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b2ccbf9f4214610b14f31573df796b4. Latest consensus state: current_term: 1 leader_uuid: "0b2ccbf9f4214610b14f31573df796b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b2ccbf9f4214610b14f31573df796b4" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 38675 } } }
I20260929 16:41:46.222106 4439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b2ccbf9f4214610b14f31573df796b4 [sys.catalog]: This master's current role is: LEADER
I20260929 16:41:46.222143 4440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b2ccbf9f4214610b14f31573df796b4 [sys.catalog]: This master's current role is: LEADER
I20260929 16:41:46.228307 4445 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:41:46.234457 4445 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:41:46.245023 4445 catalog_manager.cc:1347] Generated new cluster ID: aa0fbdfb7a81465db8c05847ab1ccc0f
I20260929 16:41:46.245178 4445 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:41:46.260306 4445 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:41:46.262421 4445 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:41:46.270066 4445 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b2ccbf9f4214610b14f31573df796b4: Generated new TSK 0
I20260929 16:41:46.271431 4445 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.197.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab.
Entry for principal kudu/127.0.197.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab.
I20260929 16:41:46.340006 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.65:0
--local_ip_for_outbound_sockets=127.0.197.65
--webserver_interface=127.0.197.65
--webserver_port=0
--tserver_master_addrs=127.0.197.126:38675
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab
--principal=kudu/127.0.197.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:35741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:41:46.702260 4459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:41:46.702586 4459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:41:46.702760 4459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:41:46.710407 4459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:41:46.710552 4459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:41:46.710745 4459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.65
I20260929 16:41:46.721541 4459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:35741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab
--principal=kudu/127.0.197.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.197.65
--webserver_port=0
--tserver_master_addrs=127.0.197.126:38675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.65
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:41:46.722931 4459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:41:46.724447 4459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:41:46.738844 4467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:46.738896 4464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:46.738929 4465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:41:46.739711 4459 server_base.cc:1034] running on GCE node
I20260929 16:41:46.740413 4459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:41:46.741451 4459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:41:46.742707 4459 hybrid_clock.cc:648] HybridClock initialized: now 1790700106742656 us; error 46 us; skew 500 ppm
I20260929 16:41:46.743134 4459 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:41:46 dist-test-slave-l31f krb5kdc[4353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700106, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:41:46.746336 4459 init.cc:377] Logged in from keytab as kudu/127.0.197.65@KRBTEST.COM (short username kudu)
I20260929 16:41:46.748020 4459 webserver.cc:460] Webserver started at http://127.0.197.65:35609/ using document root <none> and password file <none>
I20260929 16:41:46.748705 4459 fs_manager.cc:359] Metadata directory not provided
I20260929 16:41:46.748817 4459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:41:46.749076 4459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:41:46.752339 4459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/ts-0/data/instance:
uuid: "51bf7b8bddb64544bc55c556874c27ae"
format_stamp: "Formatted at 2026-09-29 16:41:46 on dist-test-slave-l31f"
server_key: "b192246bf1a1347c61695d1018ec78a2"
server_key_iv: "101481ddb732dcb8fd857fd1f1dc8350"
server_key_version: "encryptionkey@0"
I20260929 16:41:46.753118 4459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/ts-0/wal/instance:
uuid: "51bf7b8bddb64544bc55c556874c27ae"
format_stamp: "Formatted at 2026-09-29 16:41:46 on dist-test-slave-l31f"
server_key: "b192246bf1a1347c61695d1018ec78a2"
server_key_iv: "101481ddb732dcb8fd857fd1f1dc8350"
server_key_version: "encryptionkey@0"
I20260929 16:41:46.758798 4459 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260929 16:41:46.763175 4474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:46.764864 4459 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:41:46.765030 4459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/ts-0/wal
uuid: "51bf7b8bddb64544bc55c556874c27ae"
format_stamp: "Formatted at 2026-09-29 16:41:46 on dist-test-slave-l31f"
server_key: "b192246bf1a1347c61695d1018ec78a2"
server_key_iv: "101481ddb732dcb8fd857fd1f1dc8350"
server_key_version: "encryptionkey@0"
I20260929 16:41:46.765220 4459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:41:46.790235 4459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:41:46.795485 4459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:41:46.795805 4459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:41:46.796846 4459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:41:46.798285 4459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:41:46.798362 4459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:46.798514 4459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:41:46.798574 4459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:46.833799 4459 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.65:39953
I20260929 16:41:46.833853 4586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.65:39953 every 8 connection(s)
I20260929 16:41:46.835253 4459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
I20260929 16:41:46.835741 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 4459
I20260929 16:41:46.836006 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/ts-0/wal/instance
I20260929 16:41:46.836798 789 external_mini_cluster.cc:1428] Setting key 9bb80e41db8b1e564b43773a32c65288
Sep 29 16:41:46 dist-test-slave-l31f krb5kdc[4353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700106, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.65@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.197.66@KRBTEST.COM; defaulting to no policy
I20260929 16:41:46.860093 4589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:46.838022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:40995 (local address 127.0.197.126:38675)
0929 16:41:46.838474 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:46.838478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:46.839771 (+ 1293us) server_negotiation.cc:407] Connection header received
0929 16:41:46.841668 (+ 1897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:46.841677 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:46.841776 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:46.841905 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:46.843597 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:46.844397 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:46.847227 (+ 2830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:46.847633 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:46.847747 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:46.852254 (+ 4507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:46.852265 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:46.852273 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:46.852324 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:46.856040 (+ 3716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:46.856926 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:46.856935 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:46.856941 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:46.857100 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:46.857621 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:46.857629 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:46.857635 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:46.857962 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:46.858095 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:46.859305 (+ 1210us) server_negotiation.cc:299] Negotiation successful
0929 16:41:46.859801 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":248,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.66@KRBTEST.COM" created.
I20260929 16:41:46.860754 4590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:46.838678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:38675 (local address 127.0.197.65:40995)
0929 16:41:46.839596 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:46.839649 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:46.841370 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:46.842130 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:46.842139 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:46.842692 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:41:46.843401 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:46.843423 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:46.844593 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:46.844600 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:46.847041 (+ 2441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:46.847062 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:46.847889 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:46.847898 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:46.848248 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:46.849049 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:41:46.849083 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:41:46.851948 (+ 2865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:41:46.856246 (+ 4298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:46.856256 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:46.856273 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:46.856759 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:46.857226 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:46.857230 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:46.857232 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:46.857475 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:46.858125 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:41:46.858132 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:41:46.858538 (+ 406us) client_negotiation.cc:769] Sending connection context
0929 16:41:46.858828 (+ 290us) client_negotiation.cc:240] Negotiation successful
0929 16:41:46.859413 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":416,"threads_started":1}
I20260929 16:41:46.862794 4587 heartbeater.cc:347] Connected to a master server at 127.0.197.126:38675
I20260929 16:41:46.863195 4587 heartbeater.cc:464] Registering TS with master...
I20260929 16:41:46.863981 4587 heartbeater.cc:511] Master 127.0.197.126:38675 requested a full tablet report, sending...
I20260929 16:41:46.866206 4398 ts_manager.cc:194] Registered new tserver with Master: 51bf7b8bddb64544bc55c556874c27ae (127.0.197.65:39953)
I20260929 16:41:46.867779 4398 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.65@KRBTEST.COM'} at 127.0.197.65:40995
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab.
Entry for principal kudu/127.0.197.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab.
I20260929 16:41:46.886731 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.66:0
--local_ip_for_outbound_sockets=127.0.197.66
--webserver_interface=127.0.197.66
--webserver_port=0
--tserver_master_addrs=127.0.197.126:38675
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab
--principal=kudu/127.0.197.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:35741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:41:47.238276 4593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:41:47.238607 4593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:41:47.238745 4593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:41:47.246634 4593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:41:47.246827 4593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:41:47.247089 4593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.66
I20260929 16:41:47.257695 4593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:35741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab
--principal=kudu/127.0.197.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.197.66
--webserver_port=0
--tserver_master_addrs=127.0.197.126:38675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.66
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:41:47.259142 4593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:41:47.260568 4593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:41:47.275252 4599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:47.275322 4601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:47.275309 4598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:41:47.276000 4593 server_base.cc:1034] running on GCE node
I20260929 16:41:47.276909 4593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:41:47.278069 4593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:41:47.279327 4593 hybrid_clock.cc:648] HybridClock initialized: now 1790700107279254 us; error 62 us; skew 500 ppm
I20260929 16:41:47.279718 4593 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:41:47 dist-test-slave-l31f krb5kdc[4353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:41:47.283171 4593 init.cc:377] Logged in from keytab as kudu/127.0.197.66@KRBTEST.COM (short username kudu)
I20260929 16:41:47.284945 4593 webserver.cc:460] Webserver started at http://127.0.197.66:37813/ using document root <none> and password file <none>
I20260929 16:41:47.285713 4593 fs_manager.cc:359] Metadata directory not provided
I20260929 16:41:47.285868 4593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:41:47.286196 4593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:41:47.289400 4593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/ts-1/data/instance:
uuid: "950fe6c1b89a4354bda4855a0dc8dd9a"
format_stamp: "Formatted at 2026-09-29 16:41:47 on dist-test-slave-l31f"
server_key: "56498933d5e3515a3db22a4f0df904b2"
server_key_iv: "6c032982a17f0310782922116d93f2c2"
server_key_version: "encryptionkey@0"
I20260929 16:41:47.290266 4593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/ts-1/wal/instance:
uuid: "950fe6c1b89a4354bda4855a0dc8dd9a"
format_stamp: "Formatted at 2026-09-29 16:41:47 on dist-test-slave-l31f"
server_key: "56498933d5e3515a3db22a4f0df904b2"
server_key_iv: "6c032982a17f0310782922116d93f2c2"
server_key_version: "encryptionkey@0"
I20260929 16:41:47.296154 4593 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 16:41:47.300369 4608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:47.301914 4593 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 16:41:47.302124 4593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/ts-1/wal
uuid: "950fe6c1b89a4354bda4855a0dc8dd9a"
format_stamp: "Formatted at 2026-09-29 16:41:47 on dist-test-slave-l31f"
server_key: "56498933d5e3515a3db22a4f0df904b2"
server_key_iv: "6c032982a17f0310782922116d93f2c2"
server_key_version: "encryptionkey@0"
I20260929 16:41:47.302354 4593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:41:47.317044 4593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:41:47.321693 4593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:41:47.322105 4593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:41:47.323249 4593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:41:47.324747 4593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:41:47.324826 4593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:47.324972 4593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:41:47.325028 4593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:47.360179 4593 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.66:35979
I20260929 16:41:47.360245 4720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.66:35979 every 8 connection(s)
I20260929 16:41:47.361814 4593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
I20260929 16:41:47.368404 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 4593
I20260929 16:41:47.368640 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/ts-1/wal/instance
I20260929 16:41:47.369059 789 external_mini_cluster.cc:1428] Setting key 7c63a319ffc97b701798006527d32e98
Sep 29 16:41:47 dist-test-slave-l31f krb5kdc[4353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.66@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:41:47.384032 4723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:47.364719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:47239 (local address 127.0.197.126:38675)
0929 16:41:47.365160 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:47.365164 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:47.366194 (+ 1030us) server_negotiation.cc:407] Connection header received
0929 16:41:47.368054 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:47.368060 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:47.368154 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:47.368293 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:47.370220 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:47.371241 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:47.372967 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:47.373727 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:47.373881 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:47.377904 (+ 4023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:47.377909 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:47.377913 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:47.377954 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:47.380687 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:47.381533 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:47.381540 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:47.381543 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:47.381678 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:47.382128 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:47.382133 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:47.382136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:47.382412 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:47.382592 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:47.383382 (+ 790us) server_negotiation.cc:299] Negotiation successful
0929 16:41:47.383623 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":213,"threads_started":1}
I20260929 16:41:47.385098 4724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:47.365292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:38675 (local address 127.0.197.66:47239)
0929 16:41:47.366024 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:47.366069 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:47.367744 (+ 1675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:47.368645 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:47.368657 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:47.369288 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:41:47.370031 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:47.370059 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:47.371417 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:47.371422 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:47.372783 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:47.372798 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:47.373975 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:47.373980 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:47.374200 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:47.374951 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:41:47.374977 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:41:47.377619 (+ 2642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:41:47.380880 (+ 3261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:47.380891 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:47.380912 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:47.381390 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:47.381795 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:47.381800 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:47.381804 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:47.382008 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:47.382540 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:41:47.382549 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:41:47.382943 (+ 394us) client_negotiation.cc:769] Sending connection context
0929 16:41:47.383295 (+ 352us) client_negotiation.cc:240] Negotiation successful
0929 16:41:47.383902 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":284,"threads_started":1}
I20260929 16:41:47.386696 4721 heartbeater.cc:347] Connected to a master server at 127.0.197.126:38675
I20260929 16:41:47.387138 4721 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.197.67@KRBTEST.COM; defaulting to no policy
I20260929 16:41:47.388161 4721 heartbeater.cc:511] Master 127.0.197.126:38675 requested a full tablet report, sending...
I20260929 16:41:47.389961 4398 ts_manager.cc:194] Registered new tserver with Master: 950fe6c1b89a4354bda4855a0dc8dd9a (127.0.197.66:35979)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.67@KRBTEST.COM" created.
I20260929 16:41:47.390982 4398 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.66@KRBTEST.COM'} at 127.0.197.66:47239
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab.
Entry for principal kudu/127.0.197.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab.
I20260929 16:41:47.416647 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.67:0
--local_ip_for_outbound_sockets=127.0.197.67
--webserver_interface=127.0.197.67
--webserver_port=0
--tserver_master_addrs=127.0.197.126:38675
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab
--principal=kudu/127.0.197.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:35741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:41:47.765796 4727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:41:47.766170 4727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:41:47.766336 4727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:41:47.773919 4727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:41:47.774120 4727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:41:47.774308 4727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.67
I20260929 16:41:47.784832 4727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:35741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab
--principal=kudu/127.0.197.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.197.67
--webserver_port=0
--tserver_master_addrs=127.0.197.126:38675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.67
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:41:47.786288 4727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:41:47.787774 4727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:41:47.802484 4732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:47.802517 4733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:47.802567 4735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:41:47.803233 4727 server_base.cc:1034] running on GCE node
I20260929 16:41:47.803972 4727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:41:47.805097 4727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:41:47.806358 4727 hybrid_clock.cc:648] HybridClock initialized: now 1790700107806301 us; error 52 us; skew 500 ppm
I20260929 16:41:47.806717 4727 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:41:47 dist-test-slave-l31f krb5kdc[4353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:41:47.817209 4727 init.cc:377] Logged in from keytab as kudu/127.0.197.67@KRBTEST.COM (short username kudu)
I20260929 16:41:47.819067 4727 webserver.cc:460] Webserver started at http://127.0.197.67:44315/ using document root <none> and password file <none>
I20260929 16:41:47.819859 4727 fs_manager.cc:359] Metadata directory not provided
I20260929 16:41:47.820022 4727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:41:47.820413 4727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:41:47.823832 4727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/ts-2/data/instance:
uuid: "aff9b70faa8b4965875097c8dbfc2184"
format_stamp: "Formatted at 2026-09-29 16:41:47 on dist-test-slave-l31f"
server_key: "a68d00c40598416f6ad531ff3fcdc258"
server_key_iv: "e1ae1710fb8a6a696f1dc98b1ab1cbb5"
server_key_version: "encryptionkey@0"
I20260929 16:41:47.824729 4727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/ts-2/wal/instance:
uuid: "aff9b70faa8b4965875097c8dbfc2184"
format_stamp: "Formatted at 2026-09-29 16:41:47 on dist-test-slave-l31f"
server_key: "a68d00c40598416f6ad531ff3fcdc258"
server_key_iv: "e1ae1710fb8a6a696f1dc98b1ab1cbb5"
server_key_version: "encryptionkey@0"
I20260929 16:41:47.831148 4727 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 16:41:47.835696 4742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:47.837421 4727 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:41:47.837643 4727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/ts-2/wal
uuid: "aff9b70faa8b4965875097c8dbfc2184"
format_stamp: "Formatted at 2026-09-29 16:41:47 on dist-test-slave-l31f"
server_key: "a68d00c40598416f6ad531ff3fcdc258"
server_key_iv: "e1ae1710fb8a6a696f1dc98b1ab1cbb5"
server_key_version: "encryptionkey@0"
I20260929 16:41:47.837916 4727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:41:47.860140 4727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:41:47.865396 4727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:41:47.865761 4727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:41:47.867180 4727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:41:47.869429 4727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:41:47.869529 4727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:47.869673 4727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:41:47.869722 4727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:47.871093 4587 heartbeater.cc:503] Master 127.0.197.126:38675 was elected leader, sending a full tablet report...
I20260929 16:41:47.905824 4727 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.67:43595
I20260929 16:41:47.905936 4854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.67:43595 every 8 connection(s)
I20260929 16:41:47.907377 4727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
I20260929 16:41:47.908952 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 4727
I20260929 16:41:47.909162 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790700068255611-789-0/minicluster-data/ts-2/wal/instance
I20260929 16:41:47.909726 789 external_mini_cluster.cc:1428] Setting key 8ca72aee2fb26b4540ff1bd515e7e872
Sep 29 16:41:47 dist-test-slave-l31f krb5kdc[4353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.67@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:41:47.930071 4857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:47.910849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:40707 (local address 127.0.197.126:38675)
0929 16:41:47.911304 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:47.911308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:47.912497 (+ 1189us) server_negotiation.cc:407] Connection header received
0929 16:41:47.914476 (+ 1979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:47.914480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:47.914573 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:47.914675 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:47.916494 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:47.917312 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:47.919149 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:47.919609 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:47.919748 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:47.923788 (+ 4040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:47.923795 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:47.923799 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:47.923838 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:47.926852 (+ 3014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:47.927824 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:47.927827 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:47.927830 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:47.927987 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:47.928363 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:47.928366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:47.928368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:47.928649 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:47.928847 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:47.929604 (+ 757us) server_negotiation.cc:299] Negotiation successful
0929 16:41:47.929778 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":251,"threads_started":1}
I20260929 16:41:47.931195 4858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:47.911426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:38675 (local address 127.0.197.67:40707)
0929 16:41:47.912321 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:47.912376 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:47.914179 (+ 1803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:47.915000 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:47.915048 (+ 48us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:47.915665 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:41:47.916277 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:47.916309 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:47.917521 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:47.917526 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:47.918888 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:47.918903 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:47.919916 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:47.919922 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:47.920176 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:47.920954 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:41:47.920984 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:41:47.923494 (+ 2510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:41:47.927108 (+ 3614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:47.927116 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:47.927133 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:47.927704 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:47.928078 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:47.928082 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:47.928084 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:47.928263 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:47.928860 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:41:47.928867 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:41:47.929245 (+ 378us) client_negotiation.cc:769] Sending connection context
0929 16:41:47.929511 (+ 266us) client_negotiation.cc:240] Negotiation successful
0929 16:41:47.929954 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":380,"threads_started":1}
I20260929 16:41:47.932946 4855 heartbeater.cc:347] Connected to a master server at 127.0.197.126:38675
I20260929 16:41:47.933322 4855 heartbeater.cc:464] Registering TS with master...
I20260929 16:41:47.934062 4855 heartbeater.cc:511] Master 127.0.197.126:38675 requested a full tablet report, sending...
I20260929 16:41:47.935798 4398 ts_manager.cc:194] Registered new tserver with Master: aff9b70faa8b4965875097c8dbfc2184 (127.0.197.67:43595)
I20260929 16:41:47.936794 4398 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.67@KRBTEST.COM'} at 127.0.197.67:40707
I20260929 16:41:47.938305 789 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 16:41:47 dist-test-slave-l31f krb5kdc[4353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 16:41:47 dist-test-slave-l31f krb5kdc[4353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.65@KRBTEST.COM
I20260929 16:41:47.994931 4863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:47.976661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37840 (local address 127.0.197.65:39953)
0929 16:41:47.977402 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:47.977410 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:47.977441 (+ 31us) server_negotiation.cc:407] Connection header received
0929 16:41:47.977506 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:47.977513 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:47.977753 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:47.977935 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:47.980119 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:47.981242 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:47.982767 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:47.983500 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:47.983614 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:47.988543 (+ 4929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:47.988560 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:47.988577 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:47.988625 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:47.991700 (+ 3075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:47.992483 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:47.992491 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:47.992494 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:47.992646 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:47.993040 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:47.993044 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:47.993046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:47.993414 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:47.993655 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:47.994233 (+ 578us) server_negotiation.cc:299] Negotiation successful
0929 16:41:47.994556 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":301,"threads_started":1}
Sep 29 16:41:48 dist-test-slave-l31f krb5kdc[4353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:41:48.017725 4857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:48.000268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34816 (local address 127.0.197.126:38675)
0929 16:41:48.000520 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:48.000528 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:48.000709 (+ 181us) server_negotiation.cc:407] Connection header received
0929 16:41:48.000970 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:48.000976 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:48.001054 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:48.001162 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:48.002510 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:48.003541 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:48.005116 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:48.005567 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:48.005734 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:48.011178 (+ 5444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:48.011185 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:48.011189 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:48.011240 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:48.014333 (+ 3093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:48.015492 (+ 1159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:48.015503 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:48.015506 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:48.015670 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:48.016100 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:48.016105 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:48.016108 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:48.016488 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:48.016793 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:48.017113 (+ 320us) server_negotiation.cc:299] Negotiation successful
0929 16:41:48.017383 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260929 16:41:48.018409 4398 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34816
I20260929 16:41:48.019630 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 4459
I20260929 16:41:48.034157 4863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:48.020694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37852 (local address 127.0.197.65:39953)
0929 16:41:48.020965 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:48.020971 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:48.021244 (+ 273us) server_negotiation.cc:407] Connection header received
0929 16:41:48.021480 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:48.021485 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:48.021591 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:48.021711 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:48.023121 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:48.023968 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:48.025236 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:48.025717 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:48.025867 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:48.028607 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:48.028613 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:48.028616 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:48.028657 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:48.031494 (+ 2837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:48.032237 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:48.032243 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:48.032247 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:48.032385 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:48.032763 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:48.032770 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:48.032773 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:48.033036 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:48.033288 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:48.033605 (+ 317us) server_negotiation.cc:299] Negotiation successful
0929 16:41:48.033839 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260929 16:41:48.034766 4582 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:41:48.119370 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 4459
I20260929 16:41:48.136802 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 4593
Sep 29 16:41:48 dist-test-slave-l31f krb5kdc[4353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.66@KRBTEST.COM
I20260929 16:41:48.153535 4868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:48.137725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37620 (local address 127.0.197.66:35979)
0929 16:41:48.138206 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:48.138213 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:48.138237 (+ 24us) server_negotiation.cc:407] Connection header received
0929 16:41:48.138296 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:48.138302 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:48.138526 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:48.138699 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:48.139917 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:48.140899 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:48.142284 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:48.142937 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:48.143112 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:48.147400 (+ 4288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:48.147414 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:48.147437 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:48.147479 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:48.150466 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:48.151340 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:48.151348 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:48.151351 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:48.151490 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:48.151884 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:48.151888 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:48.151889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:48.152250 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:48.152588 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:48.152873 (+ 285us) server_negotiation.cc:299] Negotiation successful
0929 16:41:48.153155 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":238,"threads_started":1}
I20260929 16:41:48.155098 4716 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:41:48.240615 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 4593
I20260929 16:41:48.258666 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 4727
Sep 29 16:41:48 dist-test-slave-l31f krb5kdc[4353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.67@KRBTEST.COM
I20260929 16:41:48.277733 4870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:48.259653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34996 (local address 127.0.197.67:43595)
0929 16:41:48.260313 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:48.260320 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:48.260351 (+ 31us) server_negotiation.cc:407] Connection header received
0929 16:41:48.260412 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:48.260418 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:48.260645 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:48.260782 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:48.262309 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:48.263687 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:48.265175 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:48.265771 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:48.265927 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:48.271347 (+ 5420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:48.271364 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:48.271382 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:48.271425 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:48.274535 (+ 3110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:48.275584 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:48.275592 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:48.275595 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:48.275749 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:48.276130 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:48.276134 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:48.276136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:48.276520 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:48.276785 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:48.277092 (+ 307us) server_negotiation.cc:299] Negotiation successful
0929 16:41:48.277318 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":350,"threads_started":1}
I20260929 16:41:48.279320 4850 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:41:48.365620 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 4727
I20260929 16:41:48.384325 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 4366
I20260929 16:41:48.385558 4428 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:41:48.495066 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 4366
2026-09-29T16:41:48Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4191 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:41:48 dist-test-slave-l31f krb5kdc[4875](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:41:48 dist-test-slave-l31f krb5kdc[4875](info): set up 2 sockets
Sep 29 16:41:48 dist-test-slave-l31f krb5kdc[4875](info): commencing operation
krb5kdc: starting...
W20260929 16:41:49.611663 789 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.008s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/krb5kdc/test-user.keytab.
Sep 29 16:41:49 dist-test-slave-l31f krb5kdc[4875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:41:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:41:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.197.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
I20260929 16:41:49.827571 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:35737
--webserver_interface=127.0.197.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:43289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.126:35737
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:41:50.179821 4888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:41:50.180169 4888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:41:50.180279 4888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:41:50.187930 4888 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:41:50.188073 4888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:41:50.188122 4888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:41:50.188172 4888 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:41:50.188212 4888 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:41:50.198884 4888 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:43289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.197.126:35737
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:35737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.197.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:41:50.200439 4888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:41:50.201895 4888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:41:50.214775 4893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:50.214854 4896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:50.214784 4894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:41:50.215719 4888 server_base.cc:1034] running on GCE node
I20260929 16:41:50.217160 4888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:41:50.219290 4888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:41:50.220597 4888 hybrid_clock.cc:648] HybridClock initialized: now 1790700110220523 us; error 68 us; skew 500 ppm
I20260929 16:41:50.221052 4888 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:41:50 dist-test-slave-l31f krb5kdc[4875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:41:50.224577 4888 init.cc:377] Logged in from keytab as kudu/127.0.197.126@KRBTEST.COM (short username kudu)
I20260929 16:41:50.225942 4888 fs_manager.cc:359] Metadata directory not provided
I20260929 16:41:50.226101 4888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:41:50.226455 4888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:41:50.229947 4888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/master-0/data/instance:
uuid: "774eb4c109624cd3ba15181cdbd4f348"
format_stamp: "Formatted at 2026-09-29 16:41:50 on dist-test-slave-l31f"
server_key: "f0c5fb4cb52b6b6b078c7c3f2633cc94"
server_key_iv: "c06de19434c812300ab3fc560cdf0b63"
server_key_version: "encryptionkey@0"
I20260929 16:41:50.230841 4888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/master-0/wal/instance:
uuid: "774eb4c109624cd3ba15181cdbd4f348"
format_stamp: "Formatted at 2026-09-29 16:41:50 on dist-test-slave-l31f"
server_key: "f0c5fb4cb52b6b6b078c7c3f2633cc94"
server_key_iv: "c06de19434c812300ab3fc560cdf0b63"
server_key_version: "encryptionkey@0"
I20260929 16:41:50.237198 4888 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.003s
I20260929 16:41:50.241540 4902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:50.243256 4888 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:41:50.243507 4888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "774eb4c109624cd3ba15181cdbd4f348"
format_stamp: "Formatted at 2026-09-29 16:41:50 on dist-test-slave-l31f"
server_key: "f0c5fb4cb52b6b6b078c7c3f2633cc94"
server_key_iv: "c06de19434c812300ab3fc560cdf0b63"
server_key_version: "encryptionkey@0"
I20260929 16:41:50.243788 4888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:41:50.274230 4888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:41:50.279176 4888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:41:50.279469 4888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:41:50.299855 4888 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.126:35737
I20260929 16:41:50.299835 4953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.126:35737 every 8 connection(s)
I20260929 16:41:50.300863 4888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:41:50.305718 4954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:41:50.310031 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 4888
I20260929 16:41:50.310297 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/master-0/wal/instance
I20260929 16:41:50.311151 789 external_mini_cluster.cc:1428] Setting key daefd1669f0141412da656150c19e6be
I20260929 16:41:50.318795 4954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 774eb4c109624cd3ba15181cdbd4f348: Bootstrap starting.
I20260929 16:41:50.322257 4954 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 774eb4c109624cd3ba15181cdbd4f348: Neither blocks nor log segments found. Creating new log.
I20260929 16:41:50.323508 4954 log.cc:824] T 00000000000000000000000000000000 P 774eb4c109624cd3ba15181cdbd4f348: Log is configured to *not* fsync() on all Append() calls
I20260929 16:41:50.326604 4954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 774eb4c109624cd3ba15181cdbd4f348: No bootstrap required, opened a new log
Sep 29 16:41:50 dist-test-slave-l31f krb5kdc[4875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:41:50.337450 4954 raft_consensus.cc:348] T 00000000000000000000000000000000 P 774eb4c109624cd3ba15181cdbd4f348 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "774eb4c109624cd3ba15181cdbd4f348" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 35737 } }
I20260929 16:41:50.337960 4954 raft_consensus.cc:374] T 00000000000000000000000000000000 P 774eb4c109624cd3ba15181cdbd4f348 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:41:50.338042 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:50.313621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57914 (local address 127.0.197.126:35737)
0929 16:41:50.314793 (+ 1172us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:50.314812 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:50.314881 (+ 69us) server_negotiation.cc:407] Connection header received
0929 16:41:50.317333 (+ 2452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:50.317366 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:50.317897 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:50.318346 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:50.320022 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:50.321460 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:50.322922 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:50.323798 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:50.324007 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:50.328501 (+ 4494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:50.328513 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:50.328531 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:50.328576 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:50.332764 (+ 4188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:50.333651 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:50.333666 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:50.333677 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:50.333904 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:50.334327 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:50.334333 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:50.334336 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:50.335256 (+ 920us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:50.335581 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:50.336014 (+ 433us) server_negotiation.cc:299] Negotiation successful
0929 16:41:50.336575 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":847,"thread_start_us":556,"threads_started":1}
I20260929 16:41:50.338095 4954 raft_consensus.cc:729] T 00000000000000000000000000000000 P 774eb4c109624cd3ba15181cdbd4f348 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 774eb4c109624cd3ba15181cdbd4f348, State: Initialized, Role: FOLLOWER
I20260929 16:41:50.338721 4954 consensus_queue.cc:260] T 00000000000000000000000000000000 P 774eb4c109624cd3ba15181cdbd4f348 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "774eb4c109624cd3ba15181cdbd4f348" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 35737 } }
I20260929 16:41:50.339016 4954 raft_consensus.cc:388] T 00000000000000000000000000000000 P 774eb4c109624cd3ba15181cdbd4f348 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:41:50.339180 4954 raft_consensus.cc:482] T 00000000000000000000000000000000 P 774eb4c109624cd3ba15181cdbd4f348 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:41:50.339390 4954 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 774eb4c109624cd3ba15181cdbd4f348 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:41:50.341869 4954 raft_consensus.cc:504] T 00000000000000000000000000000000 P 774eb4c109624cd3ba15181cdbd4f348 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "774eb4c109624cd3ba15181cdbd4f348" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 35737 } }
I20260929 16:41:50.342422 4954 leader_election.cc:302] T 00000000000000000000000000000000 P 774eb4c109624cd3ba15181cdbd4f348 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 774eb4c109624cd3ba15181cdbd4f348; no voters:
I20260929 16:41:50.343106 4954 leader_election.cc:288] T 00000000000000000000000000000000 P 774eb4c109624cd3ba15181cdbd4f348 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:41:50.343255 4959 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 774eb4c109624cd3ba15181cdbd4f348 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:41:50.343974 4959 raft_consensus.cc:686] T 00000000000000000000000000000000 P 774eb4c109624cd3ba15181cdbd4f348 [term 1 LEADER]: Becoming Leader. State: Replica: 774eb4c109624cd3ba15181cdbd4f348, State: Running, Role: LEADER
I20260929 16:41:50.344521 4959 consensus_queue.cc:237] T 00000000000000000000000000000000 P 774eb4c109624cd3ba15181cdbd4f348 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "774eb4c109624cd3ba15181cdbd4f348" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 35737 } }
I20260929 16:41:50.345362 4954 sys_catalog.cc:564] T 00000000000000000000000000000000 P 774eb4c109624cd3ba15181cdbd4f348 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:41:50.347615 4960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 774eb4c109624cd3ba15181cdbd4f348 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "774eb4c109624cd3ba15181cdbd4f348" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "774eb4c109624cd3ba15181cdbd4f348" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 35737 } } }
I20260929 16:41:50.348120 4960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 774eb4c109624cd3ba15181cdbd4f348 [sys.catalog]: This master's current role is: LEADER
I20260929 16:41:50.348500 4961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 774eb4c109624cd3ba15181cdbd4f348 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 774eb4c109624cd3ba15181cdbd4f348. Latest consensus state: current_term: 1 leader_uuid: "774eb4c109624cd3ba15181cdbd4f348" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "774eb4c109624cd3ba15181cdbd4f348" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 35737 } } }
I20260929 16:41:50.348985 4961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 774eb4c109624cd3ba15181cdbd4f348 [sys.catalog]: This master's current role is: LEADER
I20260929 16:41:50.349901 4967 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:41:50.356235 4967 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:41:50.365794 4967 catalog_manager.cc:1347] Generated new cluster ID: 8033ad6550b942e5831920c2b6d406de
I20260929 16:41:50.365940 4967 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:41:50.400266 4967 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:41:50.401664 4967 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:41:50.408645 4967 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 774eb4c109624cd3ba15181cdbd4f348: Generated new TSK 0
I20260929 16:41:50.409574 4967 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.197.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab.
Entry for principal kudu/127.0.197.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab.
I20260929 16:41:50.487326 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.65:0
--local_ip_for_outbound_sockets=127.0.197.65
--webserver_interface=127.0.197.65
--webserver_port=0
--tserver_master_addrs=127.0.197.126:35737
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab
--principal=kudu/127.0.197.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:43289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:41:50.867436 4980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:41:50.867774 4980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:41:50.867887 4980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:41:50.875986 4980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:41:50.876129 4980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:41:50.876256 4980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.65
I20260929 16:41:50.887552 4980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:43289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab
--principal=kudu/127.0.197.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.197.65
--webserver_port=0
--tserver_master_addrs=127.0.197.126:35737
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.65
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:41:50.888945 4980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:41:50.890270 4980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:41:50.904860 4985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:50.904862 4988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:50.904872 4986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:41:50.905738 4980 server_base.cc:1034] running on GCE node
I20260929 16:41:50.906454 4980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:41:50.907593 4980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:41:50.908849 4980 hybrid_clock.cc:648] HybridClock initialized: now 1790700110908801 us; error 47 us; skew 500 ppm
I20260929 16:41:50.909219 4980 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:41:50 dist-test-slave-l31f krb5kdc[4875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:41:50.912798 4980 init.cc:377] Logged in from keytab as kudu/127.0.197.65@KRBTEST.COM (short username kudu)
I20260929 16:41:50.914182 4980 fs_manager.cc:359] Metadata directory not provided
I20260929 16:41:50.914331 4980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:41:50.914681 4980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:41:50.917974 4980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/ts-0/data/instance:
uuid: "b8cb3928dd3e45ed9e8deb0732d00a6c"
format_stamp: "Formatted at 2026-09-29 16:41:50 on dist-test-slave-l31f"
server_key: "53d643e0be9cda91e247c1ea2d99d0a1"
server_key_iv: "a50497380d96dfa989897e001a820a72"
server_key_version: "encryptionkey@0"
I20260929 16:41:50.918910 4980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/ts-0/wal/instance:
uuid: "b8cb3928dd3e45ed9e8deb0732d00a6c"
format_stamp: "Formatted at 2026-09-29 16:41:50 on dist-test-slave-l31f"
server_key: "53d643e0be9cda91e247c1ea2d99d0a1"
server_key_iv: "a50497380d96dfa989897e001a820a72"
server_key_version: "encryptionkey@0"
I20260929 16:41:50.924759 4980 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 16:41:50.928812 4994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:50.930485 4980 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:41:50.930730 4980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/ts-0/wal
uuid: "b8cb3928dd3e45ed9e8deb0732d00a6c"
format_stamp: "Formatted at 2026-09-29 16:41:50 on dist-test-slave-l31f"
server_key: "53d643e0be9cda91e247c1ea2d99d0a1"
server_key_iv: "a50497380d96dfa989897e001a820a72"
server_key_version: "encryptionkey@0"
I20260929 16:41:50.931010 4980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:41:50.944949 4980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:41:50.950891 4980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:41:50.951305 4980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:41:50.952525 4980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:41:50.954939 4980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:41:50.955169 4980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 16:41:50.955332 4980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:41:50.955432 4980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:50.995055 4980 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.65:34957
I20260929 16:41:50.995121 5106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.65:34957 every 8 connection(s)
I20260929 16:41:50.996243 4980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
I20260929 16:41:51.004464 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 4980
I20260929 16:41:51.004695 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/ts-0/wal/instance
I20260929 16:41:51.005452 789 external_mini_cluster.cc:1428] Setting key 79fc69ca94b6f0bbc86debc007b3fa8b
Sep 29 16:41:51 dist-test-slave-l31f krb5kdc[4875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.65@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:41:51.024277 5109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:51.000157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:49523 (local address 127.0.197.126:35737)
0929 16:41:51.002328 (+ 2171us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:51.002336 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:51.003209 (+ 873us) server_negotiation.cc:407] Connection header received
0929 16:41:51.006232 (+ 3023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:51.006240 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:51.006345 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:51.006430 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:51.008997 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:51.010292 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:51.012272 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:51.013056 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:51.013226 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:51.017599 (+ 4373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:51.017605 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:51.017608 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:51.017647 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:51.020548 (+ 2901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:51.021568 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:51.021574 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:51.021576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:51.021729 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:51.022239 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:51.022249 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:51.022252 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:51.022548 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:51.022727 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:51.023601 (+ 874us) server_negotiation.cc:299] Negotiation successful
0929 16:41:51.023890 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2008,"thread_start_us":1884,"threads_started":1}
I20260929 16:41:51.025421 5110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:51.000994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:35737 (local address 127.0.197.65:49523)
0929 16:41:51.002968 (+ 1974us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:51.003061 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:51.005799 (+ 2738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:51.007098 (+ 1299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:51.007111 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:51.008058 (+ 947us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:41:51.008755 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:51.008778 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:51.010549 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:51.010554 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:51.012048 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:51.012067 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:51.013341 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:51.013347 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:51.013727 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:51.014638 (+ 911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:41:51.014684 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:41:51.017266 (+ 2582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:41:51.020793 (+ 3527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:51.020805 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:51.020833 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:51.021389 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:51.021867 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:51.021872 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:51.021876 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:51.022096 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:51.022690 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:41:51.022699 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:41:51.023185 (+ 486us) client_negotiation.cc:769] Sending connection context
0929 16:41:51.023496 (+ 311us) client_negotiation.cc:240] Negotiation successful
0929 16:41:51.024124 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1586,"thread_start_us":1367,"threads_started":1}
WARNING: no policy specified for kudu/127.0.197.66@KRBTEST.COM; defaulting to no policy
I20260929 16:41:51.027482 5107 heartbeater.cc:347] Connected to a master server at 127.0.197.126:35737
I20260929 16:41:51.027791 5107 heartbeater.cc:464] Registering TS with master...
I20260929 16:41:51.028545 5107 heartbeater.cc:511] Master 127.0.197.126:35737 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.66@KRBTEST.COM" created.
I20260929 16:41:51.031446 4919 ts_manager.cc:194] Registered new tserver with Master: b8cb3928dd3e45ed9e8deb0732d00a6c (127.0.197.65:34957)
I20260929 16:41:51.033162 4919 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.65@KRBTEST.COM'} at 127.0.197.65:49523
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab.
Entry for principal kudu/127.0.197.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab.
I20260929 16:41:51.058272 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.66:0
--local_ip_for_outbound_sockets=127.0.197.66
--webserver_interface=127.0.197.66
--webserver_port=0
--tserver_master_addrs=127.0.197.126:35737
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab
--principal=kudu/127.0.197.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:43289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:41:51.416393 5113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:41:51.416742 5113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:41:51.416846 5113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:41:51.424986 5113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:41:51.425124 5113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:41:51.425244 5113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.66
I20260929 16:41:51.435987 5113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:43289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab
--principal=kudu/127.0.197.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.197.66
--webserver_port=0
--tserver_master_addrs=127.0.197.126:35737
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.66
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:41:51.437433 5113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:41:51.438846 5113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:41:51.453011 5119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:51.453055 5118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:51.453677 5121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:41:51.453912 5113 server_base.cc:1034] running on GCE node
I20260929 16:41:51.454655 5113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:41:51.456344 5113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:41:51.457653 5113 hybrid_clock.cc:648] HybridClock initialized: now 1790700111457571 us; error 79 us; skew 500 ppm
I20260929 16:41:51.458118 5113 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:41:51 dist-test-slave-l31f krb5kdc[4875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:41:51.461581 5113 init.cc:377] Logged in from keytab as kudu/127.0.197.66@KRBTEST.COM (short username kudu)
I20260929 16:41:51.462950 5113 fs_manager.cc:359] Metadata directory not provided
I20260929 16:41:51.463133 5113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:41:51.463481 5113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:41:51.466696 5113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/ts-1/data/instance:
uuid: "cd6e50c567494547bbd07c50fee74ba9"
format_stamp: "Formatted at 2026-09-29 16:41:51 on dist-test-slave-l31f"
server_key: "ce42cf05e3f570ac32776714e7cc3ae4"
server_key_iv: "e2f6c418b1af39e32bd41d04cbcc8321"
server_key_version: "encryptionkey@0"
I20260929 16:41:51.467736 5113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/ts-1/wal/instance:
uuid: "cd6e50c567494547bbd07c50fee74ba9"
format_stamp: "Formatted at 2026-09-29 16:41:51 on dist-test-slave-l31f"
server_key: "ce42cf05e3f570ac32776714e7cc3ae4"
server_key_iv: "e2f6c418b1af39e32bd41d04cbcc8321"
server_key_version: "encryptionkey@0"
I20260929 16:41:51.473680 5113 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 16:41:51.478539 5127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:51.480353 5113 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260929 16:41:51.480628 5113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/ts-1/wal
uuid: "cd6e50c567494547bbd07c50fee74ba9"
format_stamp: "Formatted at 2026-09-29 16:41:51 on dist-test-slave-l31f"
server_key: "ce42cf05e3f570ac32776714e7cc3ae4"
server_key_iv: "e2f6c418b1af39e32bd41d04cbcc8321"
server_key_version: "encryptionkey@0"
I20260929 16:41:51.480902 5113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:41:51.502121 5113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:41:51.506856 5113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:41:51.507220 5113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:41:51.508255 5113 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:41:51.509662 5113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:41:51.509815 5113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:51.509948 5113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:41:51.510036 5113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:51.544623 5113 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.66:41429
I20260929 16:41:51.544680 5239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.66:41429 every 8 connection(s)
I20260929 16:41:51.545619 5113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
I20260929 16:41:51.551754 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 5113
I20260929 16:41:51.551997 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/ts-1/wal/instance
I20260929 16:41:51.552448 789 external_mini_cluster.cc:1428] Setting key e468e52fc9df5a86185d4d3ecde610ce
Sep 29 16:41:51 dist-test-slave-l31f krb5kdc[4875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.66@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:41:51.569983 5242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:51.548672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:36767 (local address 127.0.197.126:35737)
0929 16:41:51.549118 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:51.549125 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:51.550378 (+ 1253us) server_negotiation.cc:407] Connection header received
0929 16:41:51.552548 (+ 2170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:51.552556 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:51.552655 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:51.552793 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:51.554720 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:51.555810 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:51.558233 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:51.559079 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:51.559239 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:51.563805 (+ 4566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:51.563811 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:51.563814 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:51.563851 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:51.566484 (+ 2633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:51.567418 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:51.567425 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:51.567429 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:51.567566 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:51.568133 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:51.568137 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:51.568139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:51.568416 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:51.568598 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:51.569343 (+ 745us) server_negotiation.cc:299] Negotiation successful
0929 16:41:51.569592 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":215,"threads_started":1}
I20260929 16:41:51.571120 5243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:51.549352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:35737 (local address 127.0.197.66:36767)
0929 16:41:51.550171 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:51.550221 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:51.552213 (+ 1992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:51.553089 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:51.553102 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:51.553762 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:41:51.554514 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:51.554543 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:51.556129 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:51.556136 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:51.557995 (+ 1859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:51.558018 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:51.559333 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:51.559339 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:51.559570 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:51.560430 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:41:51.560466 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:41:51.563487 (+ 3021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:41:51.566671 (+ 3184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:51.566682 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:51.566703 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:51.567280 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:51.567689 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:51.567694 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:51.567698 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:51.567988 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:51.568561 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:41:51.568573 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:41:51.568954 (+ 381us) client_negotiation.cc:769] Sending connection context
0929 16:41:51.569241 (+ 287us) client_negotiation.cc:240] Negotiation successful
0929 16:41:51.569848 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":301,"threads_started":1}
WARNING: no policy specified for kudu/127.0.197.67@KRBTEST.COM; defaulting to no policy
I20260929 16:41:51.572764 5240 heartbeater.cc:347] Connected to a master server at 127.0.197.126:35737
I20260929 16:41:51.573124 5240 heartbeater.cc:464] Registering TS with master...
I20260929 16:41:51.573859 5240 heartbeater.cc:511] Master 127.0.197.126:35737 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.67@KRBTEST.COM" created.
I20260929 16:41:51.575594 4919 ts_manager.cc:194] Registered new tserver with Master: cd6e50c567494547bbd07c50fee74ba9 (127.0.197.66:41429)
I20260929 16:41:51.576437 4919 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.66@KRBTEST.COM'} at 127.0.197.66:36767
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab.
Entry for principal kudu/127.0.197.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab.
I20260929 16:41:51.602334 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.67:0
--local_ip_for_outbound_sockets=127.0.197.67
--webserver_interface=127.0.197.67
--webserver_port=0
--tserver_master_addrs=127.0.197.126:35737
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab
--principal=kudu/127.0.197.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:43289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:41:51.957629 5246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:41:51.958000 5246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:41:51.958175 5246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:41:51.965847 5246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:41:51.966043 5246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:41:51.966238 5246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.67
I20260929 16:41:51.976738 5246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:43289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab
--principal=kudu/127.0.197.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.197.67
--webserver_port=0
--tserver_master_addrs=127.0.197.126:35737
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.67
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:41:51.978230 5246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:41:51.979698 5246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:41:51.994858 5252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:41:51.995675 5246 server_base.cc:1034] running on GCE node
W20260929 16:41:51.995795 5254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:51.999468 5251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:41:52.000360 5246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:41:52.001437 5246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:41:52.002884 5246 hybrid_clock.cc:648] HybridClock initialized: now 1790700112002805 us; error 58 us; skew 500 ppm
I20260929 16:41:52.003315 5246 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:41:51 dist-test-slave-l31f krb5kdc[4875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:41:52.007164 5246 init.cc:377] Logged in from keytab as kudu/127.0.197.67@KRBTEST.COM (short username kudu)
I20260929 16:41:52.008708 5246 fs_manager.cc:359] Metadata directory not provided
I20260929 16:41:52.008879 5246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:41:52.009258 5246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:41:52.012832 5246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/ts-2/data/instance:
uuid: "2d327d33c3bc4611bf0048fd29351c98"
format_stamp: "Formatted at 2026-09-29 16:41:52 on dist-test-slave-l31f"
server_key: "c44c226aeb8de80b150bed6e8da2b641"
server_key_iv: "0366d0905c242e2aca63230dfabb2b64"
server_key_version: "encryptionkey@0"
I20260929 16:41:52.013895 5246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/ts-2/wal/instance:
uuid: "2d327d33c3bc4611bf0048fd29351c98"
format_stamp: "Formatted at 2026-09-29 16:41:52 on dist-test-slave-l31f"
server_key: "c44c226aeb8de80b150bed6e8da2b641"
server_key_iv: "0366d0905c242e2aca63230dfabb2b64"
server_key_version: "encryptionkey@0"
I20260929 16:41:52.020711 5246 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.003s
I20260929 16:41:52.024848 5260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:52.026448 5246 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:41:52.026767 5246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/ts-2/wal
uuid: "2d327d33c3bc4611bf0048fd29351c98"
format_stamp: "Formatted at 2026-09-29 16:41:52 on dist-test-slave-l31f"
server_key: "c44c226aeb8de80b150bed6e8da2b641"
server_key_iv: "0366d0905c242e2aca63230dfabb2b64"
server_key_version: "encryptionkey@0"
I20260929 16:41:52.027114 5246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:41:52.036957 5107 heartbeater.cc:503] Master 127.0.197.126:35737 was elected leader, sending a full tablet report...
I20260929 16:41:52.045876 5246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:41:52.051448 5246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:41:52.051769 5246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:41:52.052929 5246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:41:52.054505 5246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:41:52.054641 5246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:52.054782 5246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:41:52.054875 5246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:52.092051 5246 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.67:40533
I20260929 16:41:52.092098 5372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.67:40533 every 8 connection(s)
I20260929 16:41:52.092985 5246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
I20260929 16:41:52.094949 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 5246
I20260929 16:41:52.095191 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/ts-2/wal/instance
I20260929 16:41:52.095763 789 external_mini_cluster.cc:1428] Setting key ee660840c1a7c2213f21c744a7889c6b
Sep 29 16:41:52 dist-test-slave-l31f krb5kdc[4875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.67@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:41:52.115376 5375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:52.095708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:35481 (local address 127.0.197.126:35737)
0929 16:41:52.096161 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:52.096165 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:52.099225 (+ 3060us) server_negotiation.cc:407] Connection header received
0929 16:41:52.100020 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:52.100025 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:52.100137 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:52.100267 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:52.101891 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:52.102676 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:52.104589 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:52.105028 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:52.105176 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:52.109126 (+ 3950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:52.109131 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:52.109135 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:52.109173 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:52.112090 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:52.112936 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:52.112945 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:52.112948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:52.113071 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:52.113511 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:52.113516 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:52.113519 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:52.113796 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:52.113951 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:52.114694 (+ 743us) server_negotiation.cc:299] Negotiation successful
0929 16:41:52.114933 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":251,"threads_started":1}
I20260929 16:41:52.116523 5376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:52.096405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:35737 (local address 127.0.197.67:35481)
0929 16:41:52.097482 (+ 1077us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:52.097520 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:52.099693 (+ 2173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:52.100517 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:52.100526 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:52.101092 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:41:52.101691 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:52.101713 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:52.102860 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:52.102865 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:52.104408 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:52.104422 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:52.105278 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:52.105285 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:52.105545 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:52.106278 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:41:52.106304 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:41:52.108805 (+ 2501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:41:52.112256 (+ 3451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:52.112266 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:52.112287 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:52.112788 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:52.113182 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:41:52.113188 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:41:52.113192 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:41:52.113392 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:41:52.113930 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:41:52.113939 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:41:52.114323 (+ 384us) client_negotiation.cc:769] Sending connection context
0929 16:41:52.114597 (+ 274us) client_negotiation.cc:240] Negotiation successful
0929 16:41:52.115209 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":824,"thread_start_us":581,"threads_started":1}
I20260929 16:41:52.118170 5373 heartbeater.cc:347] Connected to a master server at 127.0.197.126:35737
I20260929 16:41:52.118507 5373 heartbeater.cc:464] Registering TS with master...
I20260929 16:41:52.119285 5373 heartbeater.cc:511] Master 127.0.197.126:35737 requested a full tablet report, sending...
I20260929 16:41:52.120863 4919 ts_manager.cc:194] Registered new tserver with Master: 2d327d33c3bc4611bf0048fd29351c98 (127.0.197.67:40533)
I20260929 16:41:52.121637 4919 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.67@KRBTEST.COM'} at 127.0.197.67:35481
I20260929 16:41:52.124182 789 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:41:52.143383 5375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:52.128847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57920 (local address 127.0.197.126:35737)
0929 16:41:52.129018 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:52.129023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:52.129499 (+ 476us) server_negotiation.cc:407] Connection header received
0929 16:41:52.129644 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:52.129648 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:52.129714 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:52.129809 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:52.131636 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:52.132484 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:52.133885 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:52.134316 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:52.134434 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:52.137607 (+ 3173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:52.137613 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:52.137616 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:52.137657 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:52.140489 (+ 2832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:52.141226 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:52.141230 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:52.141232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:52.141349 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:52.141666 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:52.141669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:52.141671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:52.142006 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:52.142202 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:52.142831 (+ 629us) server_negotiation.cc:299] Negotiation successful
0929 16:41:52.143135 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 16:41:52.148762 4919 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57920:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:41:52.156209 4919 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:41:52.209941 5387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:52.170233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45894 (local address 127.0.197.66:41429)
0929 16:41:52.200084 (+ 29851us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:52.200094 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:52.200122 (+ 28us) server_negotiation.cc:407] Connection header received
0929 16:41:52.200220 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:52.200226 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:52.200462 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:52.200692 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:41:52.201797 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:52.203619 (+ 1822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:52.207027 (+ 3408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:52.208495 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:52.208637 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:52.208824 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:52.209425 (+ 601us) server_negotiation.cc:299] Negotiation successful
0929 16:41:52.209618 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29675,"thread_start_us":30751,"threads_started":1}
I20260929 16:41:52.210033 5388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:52.170021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.66:41429 (local address 127.0.0.1:45894)
0929 16:41:52.197858 (+ 27837us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:52.198016 (+ 158us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:52.198619 (+ 603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:52.200969 (+ 2350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:52.200974 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:52.201028 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:41:52.201594 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:52.201605 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:52.203814 (+ 2209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:52.203820 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:52.206796 (+ 2976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:52.206811 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:52.208996 (+ 2185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:52.209004 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:52.209218 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:52.209266 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 16:41:52.209358 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 16:41:52.209542 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27439,"mutex_wait_us":208,"spinlock_wait_cycles":9088,"thread_start_us":30724,"threads_started":1}
I20260929 16:41:52.210198 5389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:52.171195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58974 (local address 127.0.197.67:40533)
0929 16:41:52.180479 (+ 9284us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:52.180493 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:52.199470 (+ 18977us) server_negotiation.cc:407] Connection header received
0929 16:41:52.199754 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:52.199767 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:52.200140 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:52.200542 (+ 402us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:41:52.201914 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:52.203586 (+ 1672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:52.207036 (+ 3450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:52.208356 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:52.208486 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:52.208710 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:52.209671 (+ 961us) server_negotiation.cc:299] Negotiation successful
0929 16:41:52.209853 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8987,"thread_start_us":8345,"threads_started":1}
I20260929 16:41:52.210750 5390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:52.170386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.67:40533 (local address 127.0.0.1:58974)
0929 16:41:52.199309 (+ 28923us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:52.199333 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:52.199530 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:52.200961 (+ 1431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:52.200971 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:52.201019 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:41:52.201711 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:52.201724 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:52.203814 (+ 2090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:52.203820 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:52.206795 (+ 2975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:52.206811 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:52.208871 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:52.208877 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:52.209074 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:52.209132 (+ 58us) client_negotiation.cc:769] Sending connection context
0929 16:41:52.209226 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 16:41:52.209399 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28767,"mutex_wait_us":239,"spinlock_wait_cycles":2432,"thread_start_us":26906,"threads_started":1}
I20260929 16:41:52.211058 5386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:52.171975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:34957 (local address 127.0.0.1:33344)
0929 16:41:52.201087 (+ 29112us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:52.201110 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:52.201352 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:52.202033 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:52.202037 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:52.202063 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:41:52.202398 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:52.202410 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:52.204175 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:52.204186 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:52.206078 (+ 1892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:52.206094 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:52.208269 (+ 2175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:52.208277 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:52.208494 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:52.208557 (+ 63us) client_negotiation.cc:769] Sending connection context
0929 16:41:52.208708 (+ 151us) client_negotiation.cc:240] Negotiation successful
0929 16:41:52.210583 (+ 1875us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28979,"spinlock_wait_cycles":1792,"thread_start_us":27025,"threads_started":1}
I20260929 16:41:52.211582 5391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:52.172314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33344 (local address 127.0.197.65:34957)
0929 16:41:52.196729 (+ 24415us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:52.196744 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:52.201233 (+ 4489us) server_negotiation.cc:407] Connection header received
0929 16:41:52.201557 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:52.201568 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:52.201836 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:52.202054 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:41:52.202582 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:52.203961 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:52.206294 (+ 2333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:52.207678 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:52.207843 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:52.208110 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:52.210762 (+ 2652us) server_negotiation.cc:299] Negotiation successful
0929 16:41:52.211319 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24110,"thread_start_us":24435,"threads_started":1}
I20260929 16:41:52.214548 5308 tablet_service.cc:1467] Processing CreateTablet for tablet 6221d10554a946d0b14d2a67f5dadbab (DEFAULT_TABLE table=test-table [id=bd3148002b8142b99115a95e647c774f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:41:52.214548 5175 tablet_service.cc:1467] Processing CreateTablet for tablet 6221d10554a946d0b14d2a67f5dadbab (DEFAULT_TABLE table=test-table [id=bd3148002b8142b99115a95e647c774f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:41:52.215220 5042 tablet_service.cc:1467] Processing CreateTablet for tablet 6221d10554a946d0b14d2a67f5dadbab (DEFAULT_TABLE table=test-table [id=bd3148002b8142b99115a95e647c774f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:41:52.216118 5308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6221d10554a946d0b14d2a67f5dadbab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:41:52.216121 5175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6221d10554a946d0b14d2a67f5dadbab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:41:52.216467 5042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6221d10554a946d0b14d2a67f5dadbab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:41:52.230383 5392 tablet_bootstrap.cc:492] T 6221d10554a946d0b14d2a67f5dadbab P b8cb3928dd3e45ed9e8deb0732d00a6c: Bootstrap starting.
I20260929 16:41:52.233933 5392 tablet_bootstrap.cc:654] T 6221d10554a946d0b14d2a67f5dadbab P b8cb3928dd3e45ed9e8deb0732d00a6c: Neither blocks nor log segments found. Creating new log.
I20260929 16:41:52.234925 5394 tablet_bootstrap.cc:492] T 6221d10554a946d0b14d2a67f5dadbab P 2d327d33c3bc4611bf0048fd29351c98: Bootstrap starting.
I20260929 16:41:52.235481 5392 log.cc:824] T 6221d10554a946d0b14d2a67f5dadbab P b8cb3928dd3e45ed9e8deb0732d00a6c: Log is configured to *not* fsync() on all Append() calls
I20260929 16:41:52.236074 5393 tablet_bootstrap.cc:492] T 6221d10554a946d0b14d2a67f5dadbab P cd6e50c567494547bbd07c50fee74ba9: Bootstrap starting.
I20260929 16:41:52.240203 5392 tablet_bootstrap.cc:492] T 6221d10554a946d0b14d2a67f5dadbab P b8cb3928dd3e45ed9e8deb0732d00a6c: No bootstrap required, opened a new log
I20260929 16:41:52.240532 5392 ts_tablet_manager.cc:1397] T 6221d10554a946d0b14d2a67f5dadbab P b8cb3928dd3e45ed9e8deb0732d00a6c: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260929 16:41:52.240733 5394 tablet_bootstrap.cc:654] T 6221d10554a946d0b14d2a67f5dadbab P 2d327d33c3bc4611bf0048fd29351c98: Neither blocks nor log segments found. Creating new log.
I20260929 16:41:52.241619 5393 tablet_bootstrap.cc:654] T 6221d10554a946d0b14d2a67f5dadbab P cd6e50c567494547bbd07c50fee74ba9: Neither blocks nor log segments found. Creating new log.
I20260929 16:41:52.242138 5394 log.cc:824] T 6221d10554a946d0b14d2a67f5dadbab P 2d327d33c3bc4611bf0048fd29351c98: Log is configured to *not* fsync() on all Append() calls
I20260929 16:41:52.242934 5393 log.cc:824] T 6221d10554a946d0b14d2a67f5dadbab P cd6e50c567494547bbd07c50fee74ba9: Log is configured to *not* fsync() on all Append() calls
I20260929 16:41:52.246048 5394 tablet_bootstrap.cc:492] T 6221d10554a946d0b14d2a67f5dadbab P 2d327d33c3bc4611bf0048fd29351c98: No bootstrap required, opened a new log
I20260929 16:41:52.246395 5394 ts_tablet_manager.cc:1397] T 6221d10554a946d0b14d2a67f5dadbab P 2d327d33c3bc4611bf0048fd29351c98: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260929 16:41:52.249039 5393 tablet_bootstrap.cc:492] T 6221d10554a946d0b14d2a67f5dadbab P cd6e50c567494547bbd07c50fee74ba9: No bootstrap required, opened a new log
I20260929 16:41:52.249382 5393 ts_tablet_manager.cc:1397] T 6221d10554a946d0b14d2a67f5dadbab P cd6e50c567494547bbd07c50fee74ba9: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.004s
I20260929 16:41:52.249810 5392 raft_consensus.cc:348] T 6221d10554a946d0b14d2a67f5dadbab P b8cb3928dd3e45ed9e8deb0732d00a6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6e50c567494547bbd07c50fee74ba9" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 41429 } } peers { permanent_uuid: "2d327d33c3bc4611bf0048fd29351c98" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40533 } } peers { permanent_uuid: "b8cb3928dd3e45ed9e8deb0732d00a6c" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 34957 } }
I20260929 16:41:52.250275 5392 raft_consensus.cc:374] T 6221d10554a946d0b14d2a67f5dadbab P b8cb3928dd3e45ed9e8deb0732d00a6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:41:52.250387 5392 raft_consensus.cc:729] T 6221d10554a946d0b14d2a67f5dadbab P b8cb3928dd3e45ed9e8deb0732d00a6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8cb3928dd3e45ed9e8deb0732d00a6c, State: Initialized, Role: FOLLOWER
I20260929 16:41:52.251153 5392 consensus_queue.cc:260] T 6221d10554a946d0b14d2a67f5dadbab P b8cb3928dd3e45ed9e8deb0732d00a6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6e50c567494547bbd07c50fee74ba9" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 41429 } } peers { permanent_uuid: "2d327d33c3bc4611bf0048fd29351c98" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40533 } } peers { permanent_uuid: "b8cb3928dd3e45ed9e8deb0732d00a6c" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 34957 } }
I20260929 16:41:52.253013 5392 ts_tablet_manager.cc:1428] T 6221d10554a946d0b14d2a67f5dadbab P b8cb3928dd3e45ed9e8deb0732d00a6c: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260929 16:41:52.259289 5394 raft_consensus.cc:348] T 6221d10554a946d0b14d2a67f5dadbab P 2d327d33c3bc4611bf0048fd29351c98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6e50c567494547bbd07c50fee74ba9" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 41429 } } peers { permanent_uuid: "2d327d33c3bc4611bf0048fd29351c98" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40533 } } peers { permanent_uuid: "b8cb3928dd3e45ed9e8deb0732d00a6c" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 34957 } }
I20260929 16:41:52.259850 5394 raft_consensus.cc:374] T 6221d10554a946d0b14d2a67f5dadbab P 2d327d33c3bc4611bf0048fd29351c98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:41:52.259970 5394 raft_consensus.cc:729] T 6221d10554a946d0b14d2a67f5dadbab P 2d327d33c3bc4611bf0048fd29351c98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d327d33c3bc4611bf0048fd29351c98, State: Initialized, Role: FOLLOWER
I20260929 16:41:52.260350 5393 raft_consensus.cc:348] T 6221d10554a946d0b14d2a67f5dadbab P cd6e50c567494547bbd07c50fee74ba9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6e50c567494547bbd07c50fee74ba9" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 41429 } } peers { permanent_uuid: "2d327d33c3bc4611bf0048fd29351c98" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40533 } } peers { permanent_uuid: "b8cb3928dd3e45ed9e8deb0732d00a6c" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 34957 } }
I20260929 16:41:52.260790 5393 raft_consensus.cc:374] T 6221d10554a946d0b14d2a67f5dadbab P cd6e50c567494547bbd07c50fee74ba9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:41:52.260888 5393 raft_consensus.cc:729] T 6221d10554a946d0b14d2a67f5dadbab P cd6e50c567494547bbd07c50fee74ba9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd6e50c567494547bbd07c50fee74ba9, State: Initialized, Role: FOLLOWER
I20260929 16:41:52.260684 5394 consensus_queue.cc:260] T 6221d10554a946d0b14d2a67f5dadbab P 2d327d33c3bc4611bf0048fd29351c98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6e50c567494547bbd07c50fee74ba9" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 41429 } } peers { permanent_uuid: "2d327d33c3bc4611bf0048fd29351c98" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40533 } } peers { permanent_uuid: "b8cb3928dd3e45ed9e8deb0732d00a6c" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 34957 } }
I20260929 16:41:52.261595 5393 consensus_queue.cc:260] T 6221d10554a946d0b14d2a67f5dadbab P cd6e50c567494547bbd07c50fee74ba9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6e50c567494547bbd07c50fee74ba9" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 41429 } } peers { permanent_uuid: "2d327d33c3bc4611bf0048fd29351c98" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40533 } } peers { permanent_uuid: "b8cb3928dd3e45ed9e8deb0732d00a6c" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 34957 } }
I20260929 16:41:52.262919 5240 heartbeater.cc:503] Master 127.0.197.126:35737 was elected leader, sending a full tablet report...
I20260929 16:41:52.264104 5393 ts_tablet_manager.cc:1428] T 6221d10554a946d0b14d2a67f5dadbab P cd6e50c567494547bbd07c50fee74ba9: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260929 16:41:52.265239 5373 heartbeater.cc:503] Master 127.0.197.126:35737 was elected leader, sending a full tablet report...
I20260929 16:41:52.265779 5394 ts_tablet_manager.cc:1428] T 6221d10554a946d0b14d2a67f5dadbab P 2d327d33c3bc4611bf0048fd29351c98: Time spent starting tablet: real 0.019s user 0.010s sys 0.008s
I20260929 16:41:52.282420 5398 raft_consensus.cc:482] T 6221d10554a946d0b14d2a67f5dadbab P b8cb3928dd3e45ed9e8deb0732d00a6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:41:52.282706 5398 raft_consensus.cc:504] T 6221d10554a946d0b14d2a67f5dadbab P b8cb3928dd3e45ed9e8deb0732d00a6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6e50c567494547bbd07c50fee74ba9" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 41429 } } peers { permanent_uuid: "2d327d33c3bc4611bf0048fd29351c98" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40533 } } peers { permanent_uuid: "b8cb3928dd3e45ed9e8deb0732d00a6c" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 34957 } }
I20260929 16:41:52.284633 5398 leader_election.cc:288] T 6221d10554a946d0b14d2a67f5dadbab P b8cb3928dd3e45ed9e8deb0732d00a6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd6e50c567494547bbd07c50fee74ba9 (127.0.197.66:41429), 2d327d33c3bc4611bf0048fd29351c98 (127.0.197.67:40533)
I20260929 16:41:52.291523 5389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:52.285333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:55917 (local address 127.0.197.67:40533)
0929 16:41:52.285563 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:52.285571 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:52.285971 (+ 400us) server_negotiation.cc:407] Connection header received
0929 16:41:52.286173 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:52.286178 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:52.286235 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:52.286352 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:41:52.287088 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:52.287782 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:52.289665 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:52.290454 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:52.290666 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:52.290752 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:52.291090 (+ 338us) server_negotiation.cc:299] Negotiation successful
0929 16:41:52.291230 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 16:41:52.291550 5387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:52.285070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:51627 (local address 127.0.197.66:41429)
0929 16:41:52.285276 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:52.285283 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:52.285891 (+ 608us) server_negotiation.cc:407] Connection header received
0929 16:41:52.286117 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:52.286124 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:52.286227 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:52.286326 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:41:52.286954 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:52.287731 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:52.289668 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:52.290451 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:52.290652 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:52.290736 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:52.291153 (+ 417us) server_negotiation.cc:299] Negotiation successful
0929 16:41:52.291290 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 16:41:52.291692 5401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:52.284885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.66:41429 (local address 127.0.197.65:51627)
0929 16:41:52.285752 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:52.285772 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:52.285969 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:52.286356 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:52.286361 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:52.286384 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:41:52.286802 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:52.286815 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:52.287892 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:52.287899 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:52.289497 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:52.289509 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:52.290736 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:52.290742 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:52.290899 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:52.290946 (+ 47us) client_negotiation.cc:769] Sending connection context
0929 16:41:52.291111 (+ 165us) client_negotiation.cc:240] Negotiation successful
0929 16:41:52.291241 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"mutex_wait_us":65,"thread_start_us":495,"threads_started":1}
I20260929 16:41:52.291926 5402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:52.284913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.67:40533 (local address 127.0.197.65:55917)
0929 16:41:52.285833 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:52.285879 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:52.286033 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:52.286366 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:52.286370 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:52.286394 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:41:52.286848 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:52.286857 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:52.287943 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:52.287950 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:52.289497 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:52.289509 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:52.290692 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:52.290709 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:52.290886 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:52.290928 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 16:41:52.291063 (+ 135us) client_negotiation.cc:240] Negotiation successful
0929 16:41:52.291213 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":813,"mutex_wait_us":21,"thread_start_us":768,"threads_started":1}
I20260929 16:41:52.292861 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6221d10554a946d0b14d2a67f5dadbab" candidate_uuid: "b8cb3928dd3e45ed9e8deb0732d00a6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd6e50c567494547bbd07c50fee74ba9" is_pre_election: true
I20260929 16:41:52.292912 5328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6221d10554a946d0b14d2a67f5dadbab" candidate_uuid: "b8cb3928dd3e45ed9e8deb0732d00a6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d327d33c3bc4611bf0048fd29351c98" is_pre_election: true
I20260929 16:41:52.293376 5328 raft_consensus.cc:2454] T 6221d10554a946d0b14d2a67f5dadbab P 2d327d33c3bc4611bf0048fd29351c98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8cb3928dd3e45ed9e8deb0732d00a6c in term 0.
I20260929 16:41:52.293375 5195 raft_consensus.cc:2454] T 6221d10554a946d0b14d2a67f5dadbab P cd6e50c567494547bbd07c50fee74ba9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8cb3928dd3e45ed9e8deb0732d00a6c in term 0.
I20260929 16:41:52.294064 4997 leader_election.cc:302] T 6221d10554a946d0b14d2a67f5dadbab P b8cb3928dd3e45ed9e8deb0732d00a6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8cb3928dd3e45ed9e8deb0732d00a6c, cd6e50c567494547bbd07c50fee74ba9; no voters:
I20260929 16:41:52.294483 5398 raft_consensus.cc:2781] T 6221d10554a946d0b14d2a67f5dadbab P b8cb3928dd3e45ed9e8deb0732d00a6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:41:52.294634 5398 raft_consensus.cc:482] T 6221d10554a946d0b14d2a67f5dadbab P b8cb3928dd3e45ed9e8deb0732d00a6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:41:52.294734 5398 raft_consensus.cc:3037] T 6221d10554a946d0b14d2a67f5dadbab P b8cb3928dd3e45ed9e8deb0732d00a6c [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:41:52.296957 5398 raft_consensus.cc:504] T 6221d10554a946d0b14d2a67f5dadbab P b8cb3928dd3e45ed9e8deb0732d00a6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6e50c567494547bbd07c50fee74ba9" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 41429 } } peers { permanent_uuid: "2d327d33c3bc4611bf0048fd29351c98" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40533 } } peers { permanent_uuid: "b8cb3928dd3e45ed9e8deb0732d00a6c" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 34957 } }
I20260929 16:41:52.297685 5398 leader_election.cc:288] T 6221d10554a946d0b14d2a67f5dadbab P b8cb3928dd3e45ed9e8deb0732d00a6c [CANDIDATE]: Term 1 election: Requested vote from peers cd6e50c567494547bbd07c50fee74ba9 (127.0.197.66:41429), 2d327d33c3bc4611bf0048fd29351c98 (127.0.197.67:40533)
W20260929 16:41:52.297981 5241 tablet.cc:2369] T 6221d10554a946d0b14d2a67f5dadbab P cd6e50c567494547bbd07c50fee74ba9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:41:52.298224 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6221d10554a946d0b14d2a67f5dadbab" candidate_uuid: "b8cb3928dd3e45ed9e8deb0732d00a6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd6e50c567494547bbd07c50fee74ba9"
I20260929 16:41:52.298319 5328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6221d10554a946d0b14d2a67f5dadbab" candidate_uuid: "b8cb3928dd3e45ed9e8deb0732d00a6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d327d33c3bc4611bf0048fd29351c98"
I20260929 16:41:52.298441 5195 raft_consensus.cc:3037] T 6221d10554a946d0b14d2a67f5dadbab P cd6e50c567494547bbd07c50fee74ba9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:41:52.298528 5328 raft_consensus.cc:3037] T 6221d10554a946d0b14d2a67f5dadbab P 2d327d33c3bc4611bf0048fd29351c98 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:41:52.301029 5195 raft_consensus.cc:2454] T 6221d10554a946d0b14d2a67f5dadbab P cd6e50c567494547bbd07c50fee74ba9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8cb3928dd3e45ed9e8deb0732d00a6c in term 1.
I20260929 16:41:52.301532 4997 leader_election.cc:302] T 6221d10554a946d0b14d2a67f5dadbab P b8cb3928dd3e45ed9e8deb0732d00a6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8cb3928dd3e45ed9e8deb0732d00a6c, cd6e50c567494547bbd07c50fee74ba9; no voters:
I20260929 16:41:52.301620 5328 raft_consensus.cc:2454] T 6221d10554a946d0b14d2a67f5dadbab P 2d327d33c3bc4611bf0048fd29351c98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8cb3928dd3e45ed9e8deb0732d00a6c in term 1.
I20260929 16:41:52.303236 5398 raft_consensus.cc:2781] T 6221d10554a946d0b14d2a67f5dadbab P b8cb3928dd3e45ed9e8deb0732d00a6c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:41:52.303815 5398 raft_consensus.cc:686] T 6221d10554a946d0b14d2a67f5dadbab P b8cb3928dd3e45ed9e8deb0732d00a6c [term 1 LEADER]: Becoming Leader. State: Replica: b8cb3928dd3e45ed9e8deb0732d00a6c, State: Running, Role: LEADER
I20260929 16:41:52.304337 5398 consensus_queue.cc:237] T 6221d10554a946d0b14d2a67f5dadbab P b8cb3928dd3e45ed9e8deb0732d00a6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6e50c567494547bbd07c50fee74ba9" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 41429 } } peers { permanent_uuid: "2d327d33c3bc4611bf0048fd29351c98" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40533 } } peers { permanent_uuid: "b8cb3928dd3e45ed9e8deb0732d00a6c" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 34957 } }
I20260929 16:41:52.309134 4919 catalog_manager.cc:5495] T 6221d10554a946d0b14d2a67f5dadbab P b8cb3928dd3e45ed9e8deb0732d00a6c reported cstate change: term changed from 0 to 1, leader changed from <none> to b8cb3928dd3e45ed9e8deb0732d00a6c (127.0.197.65). New cstate: current_term: 1 leader_uuid: "b8cb3928dd3e45ed9e8deb0732d00a6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd6e50c567494547bbd07c50fee74ba9" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 41429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d327d33c3bc4611bf0048fd29351c98" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8cb3928dd3e45ed9e8deb0732d00a6c" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 34957 } health_report { overall_health: HEALTHY } } }
I20260929 16:41:52.334569 5391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:52.328854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33350 (local address 127.0.197.65:34957)
0929 16:41:52.329094 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:52.329101 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:52.329207 (+ 106us) server_negotiation.cc:407] Connection header received
0929 16:41:52.329476 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:52.329480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:52.329563 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:52.329762 (+ 199us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:41:52.330134 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:52.330904 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:52.332326 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:52.332747 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:52.332893 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:52.333412 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:41:52.333962 (+ 550us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:41:52.334151 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:52.334205 (+ 54us) server_negotiation.cc:299] Negotiation successful
0929 16:41:52.334294 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 16:41:52.343884 5328 raft_consensus.cc:1261] T 6221d10554a946d0b14d2a67f5dadbab P 2d327d33c3bc4611bf0048fd29351c98 [term 1 FOLLOWER]: Refusing update from remote peer b8cb3928dd3e45ed9e8deb0732d00a6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:41:52.343891 5195 raft_consensus.cc:1261] T 6221d10554a946d0b14d2a67f5dadbab P cd6e50c567494547bbd07c50fee74ba9 [term 1 FOLLOWER]: Refusing update from remote peer b8cb3928dd3e45ed9e8deb0732d00a6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260929 16:41:52.344522 5374 tablet.cc:2369] T 6221d10554a946d0b14d2a67f5dadbab P 2d327d33c3bc4611bf0048fd29351c98: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:41:52.345047 5398 consensus_queue.cc:1035] T 6221d10554a946d0b14d2a67f5dadbab P b8cb3928dd3e45ed9e8deb0732d00a6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d327d33c3bc4611bf0048fd29351c98" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:41:52.345424 5403 consensus_queue.cc:1035] T 6221d10554a946d0b14d2a67f5dadbab P b8cb3928dd3e45ed9e8deb0732d00a6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd6e50c567494547bbd07c50fee74ba9" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 41429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:41:52.376688 4919 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:41:52.376999 4919 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:41:52.381124 4919 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 774eb4c109624cd3ba15181cdbd4f348: Sending DeleteTablet for 3 replicas of tablet 6221d10554a946d0b14d2a67f5dadbab
I20260929 16:41:52.382486 5175 tablet_service.cc:1514] Processing DeleteTablet for tablet 6221d10554a946d0b14d2a67f5dadbab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:41:52 UTC) from {username='kudu', principal='kudu/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:45894
I20260929 16:41:52.382490 5308 tablet_service.cc:1514] Processing DeleteTablet for tablet 6221d10554a946d0b14d2a67f5dadbab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:41:52 UTC) from {username='kudu', principal='kudu/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:58974
I20260929 16:41:52.382696 5042 tablet_service.cc:1514] Processing DeleteTablet for tablet 6221d10554a946d0b14d2a67f5dadbab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:41:52 UTC) from {username='kudu', principal='kudu/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:33344
I20260929 16:41:52.384786 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 4980
I20260929 16:41:52.398483 5416 tablet_replica.cc:330] T 6221d10554a946d0b14d2a67f5dadbab P cd6e50c567494547bbd07c50fee74ba9: stopping tablet replica
I20260929 16:41:52.399150 5416 raft_consensus.cc:2229] T 6221d10554a946d0b14d2a67f5dadbab P cd6e50c567494547bbd07c50fee74ba9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:41:52.399777 5417 tablet_replica.cc:330] T 6221d10554a946d0b14d2a67f5dadbab P 2d327d33c3bc4611bf0048fd29351c98: stopping tablet replica
I20260929 16:41:52.399822 5416 raft_consensus.cc:2258] T 6221d10554a946d0b14d2a67f5dadbab P cd6e50c567494547bbd07c50fee74ba9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:41:52.400329 5417 raft_consensus.cc:2229] T 6221d10554a946d0b14d2a67f5dadbab P 2d327d33c3bc4611bf0048fd29351c98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:41:52.400876 5417 raft_consensus.cc:2258] T 6221d10554a946d0b14d2a67f5dadbab P 2d327d33c3bc4611bf0048fd29351c98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:41:52.402614 5416 ts_tablet_manager.cc:1905] T 6221d10554a946d0b14d2a67f5dadbab P cd6e50c567494547bbd07c50fee74ba9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:41:52.403133 5417 ts_tablet_manager.cc:1905] T 6221d10554a946d0b14d2a67f5dadbab P 2d327d33c3bc4611bf0048fd29351c98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:41:52.410847 5417 ts_tablet_manager.cc:1918] T 6221d10554a946d0b14d2a67f5dadbab P 2d327d33c3bc4611bf0048fd29351c98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:41:52.411119 5417 log.cc:1196] T 6221d10554a946d0b14d2a67f5dadbab P 2d327d33c3bc4611bf0048fd29351c98: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/ts-2/wal/wals/6221d10554a946d0b14d2a67f5dadbab
I20260929 16:41:52.411723 5417 ts_tablet_manager.cc:1939] T 6221d10554a946d0b14d2a67f5dadbab P 2d327d33c3bc4611bf0048fd29351c98: Deleting consensus metadata
I20260929 16:41:52.411971 5418 tablet_replica.cc:330] T 6221d10554a946d0b14d2a67f5dadbab P b8cb3928dd3e45ed9e8deb0732d00a6c: stopping tablet replica
I20260929 16:41:52.412362 5416 ts_tablet_manager.cc:1918] T 6221d10554a946d0b14d2a67f5dadbab P cd6e50c567494547bbd07c50fee74ba9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:41:52.412575 5416 log.cc:1196] T 6221d10554a946d0b14d2a67f5dadbab P cd6e50c567494547bbd07c50fee74ba9: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/ts-1/wal/wals/6221d10554a946d0b14d2a67f5dadbab
I20260929 16:41:52.413165 5416 ts_tablet_manager.cc:1939] T 6221d10554a946d0b14d2a67f5dadbab P cd6e50c567494547bbd07c50fee74ba9: Deleting consensus metadata
I20260929 16:41:52.413758 4904 catalog_manager.cc:4841] TS 2d327d33c3bc4611bf0048fd29351c98 (127.0.197.67:40533): tablet 6221d10554a946d0b14d2a67f5dadbab (table test-table [id=bd3148002b8142b99115a95e647c774f]) successfully deleted
I20260929 16:41:52.414352 5418 raft_consensus.cc:2229] T 6221d10554a946d0b14d2a67f5dadbab P b8cb3928dd3e45ed9e8deb0732d00a6c [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:41:52.415024 4905 catalog_manager.cc:4841] TS cd6e50c567494547bbd07c50fee74ba9 (127.0.197.66:41429): tablet 6221d10554a946d0b14d2a67f5dadbab (table test-table [id=bd3148002b8142b99115a95e647c774f]) successfully deleted
I20260929 16:41:52.415108 5418 raft_consensus.cc:2258] T 6221d10554a946d0b14d2a67f5dadbab P b8cb3928dd3e45ed9e8deb0732d00a6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:41:52.417264 5418 ts_tablet_manager.cc:1905] T 6221d10554a946d0b14d2a67f5dadbab P b8cb3928dd3e45ed9e8deb0732d00a6c: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 16:41:52 dist-test-slave-l31f krb5kdc[4875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.65@KRBTEST.COM
I20260929 16:41:52.423954 5418 ts_tablet_manager.cc:1918] T 6221d10554a946d0b14d2a67f5dadbab P b8cb3928dd3e45ed9e8deb0732d00a6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:41:52.424181 5418 log.cc:1196] T 6221d10554a946d0b14d2a67f5dadbab P b8cb3928dd3e45ed9e8deb0732d00a6c: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790700068255611-789-0/minicluster-data/ts-0/wal/wals/6221d10554a946d0b14d2a67f5dadbab
I20260929 16:41:52.424697 5418 ts_tablet_manager.cc:1939] T 6221d10554a946d0b14d2a67f5dadbab P b8cb3928dd3e45ed9e8deb0732d00a6c: Deleting consensus metadata
I20260929 16:41:52.426517 4903 catalog_manager.cc:4841] TS b8cb3928dd3e45ed9e8deb0732d00a6c (127.0.197.65:34957): tablet 6221d10554a946d0b14d2a67f5dadbab (table test-table [id=bd3148002b8142b99115a95e647c774f]) successfully deleted
I20260929 16:41:52.430352 5391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:52.386183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33366 (local address 127.0.197.65:34957)
0929 16:41:52.386395 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:52.386401 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:52.410183 (+ 23782us) server_negotiation.cc:407] Connection header received
0929 16:41:52.410436 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:52.410442 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:52.410545 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:52.410663 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:52.412527 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:52.413845 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:52.416247 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:52.416938 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:52.417092 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:52.422547 (+ 5455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:52.422572 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:52.422600 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:52.422657 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:52.427271 (+ 4614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:52.428110 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:52.428122 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:52.428125 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:52.428319 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:52.428684 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:52.428689 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:52.428693 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:52.429184 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:52.429506 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:52.429677 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 16:41:52.429939 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 16:41:52.431195 5102 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:41:52.536173 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 4980
I20260929 16:41:52.556090 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 5113
Sep 29 16:41:52 dist-test-slave-l31f krb5kdc[4875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.66@KRBTEST.COM
I20260929 16:41:52.572556 5387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:52.557078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45898 (local address 127.0.197.66:41429)
0929 16:41:52.557300 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:52.557305 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:52.557319 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:41:52.557374 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:52.557378 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:52.557456 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:52.557578 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:52.558783 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:52.559627 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:52.560884 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:52.561309 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:52.561446 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:52.566642 (+ 5196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:52.566658 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:52.566676 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:52.566728 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:52.569640 (+ 2912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:52.570479 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:52.570487 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:52.570490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:52.570635 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:52.570999 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:52.571065 (+ 66us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:52.571071 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:52.571523 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:52.571755 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:52.572100 (+ 345us) server_negotiation.cc:299] Negotiation successful
0929 16:41:52.572298 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 16:41:52.573865 5235 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:41:52.665414 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 5113
I20260929 16:41:52.683473 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 5246
Sep 29 16:41:52 dist-test-slave-l31f krb5kdc[4875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.67@KRBTEST.COM
I20260929 16:41:52.699357 5389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:52.684259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58978 (local address 127.0.197.67:40533)
0929 16:41:52.684453 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:52.684460 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:52.684481 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:41:52.684676 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:52.684682 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:52.684779 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:52.684893 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:52.686186 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:52.686914 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:52.688204 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:52.688607 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:52.688749 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:52.693391 (+ 4642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:52.693409 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:52.693426 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:41:52.693476 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:52.696341 (+ 2865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:52.697142 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:52.697151 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:52.697153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:52.697325 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:41:52.697735 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:41:52.697743 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:41:52.697750 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:41:52.698098 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:52.698414 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:52.698751 (+ 337us) server_negotiation.cc:299] Negotiation successful
0929 16:41:52.699042 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 16:41:52.700023 5368 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:41:52.790179 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 5246
I20260929 16:41:52.806660 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 4888
I20260929 16:41:52.807932 4949 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:41:52.917112 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 4888
2026-09-29T16:41:52Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4420 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-29T16:41:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:41:52Z Disabled control of system clock
I20260929 16:41:52.988397 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:42879
--webserver_interface=127.0.197.126
--webserver_port=0
--builtin_ntp_servers=127.0.197.84:34767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.126:42879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 16:41:53.331473 5427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:41:53.331812 5427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:41:53.331969 5427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:41:53.339562 5427 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:41:53.339709 5427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:41:53.339809 5427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:41:53.339921 5427 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:41:53.339989 5427 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:41:53.350351 5427 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:34767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.197.126:42879
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:42879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.197.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:41:53.351926 5427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:41:53.353850 5427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:41:53.366835 5432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:53.366941 5433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:53.366915 5435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:41:53.367807 5427 server_base.cc:1034] running on GCE node
I20260929 16:41:53.368674 5427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:41:53.370337 5427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:41:53.371608 5427 hybrid_clock.cc:648] HybridClock initialized: now 1790700113371563 us; error 53 us; skew 500 ppm
I20260929 16:41:53.372108 5427 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:41:53.383111 5427 webserver.cc:460] Webserver started at http://127.0.197.126:42363/ using document root <none> and password file <none>
I20260929 16:41:53.384047 5427 fs_manager.cc:359] Metadata directory not provided
I20260929 16:41:53.384159 5427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:41:53.384560 5427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:41:53.387890 5427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/master-0/data/instance:
uuid: "e99dcbdc84474c229dfaf49a99ade2dc"
format_stamp: "Formatted at 2026-09-29 16:41:53 on dist-test-slave-l31f"
server_key: "0c6327e1e4827883673d7c725e5f556c"
server_key_iv: "6504f6c02502d4a601a873268a60486d"
server_key_version: "encryptionkey@0"
I20260929 16:41:53.388836 5427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/master-0/wal/instance:
uuid: "e99dcbdc84474c229dfaf49a99ade2dc"
format_stamp: "Formatted at 2026-09-29 16:41:53 on dist-test-slave-l31f"
server_key: "0c6327e1e4827883673d7c725e5f556c"
server_key_iv: "6504f6c02502d4a601a873268a60486d"
server_key_version: "encryptionkey@0"
I20260929 16:41:53.394941 5427 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 16:41:53.399214 5441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:53.400920 5427 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:41:53.401175 5427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "e99dcbdc84474c229dfaf49a99ade2dc"
format_stamp: "Formatted at 2026-09-29 16:41:53 on dist-test-slave-l31f"
server_key: "0c6327e1e4827883673d7c725e5f556c"
server_key_iv: "6504f6c02502d4a601a873268a60486d"
server_key_version: "encryptionkey@0"
I20260929 16:41:53.401490 5427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:41:53.408545 5427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:41:53.409513 5427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:41:53.409880 5427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:41:53.430413 5427 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.126:42879
I20260929 16:41:53.430435 5492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.126:42879 every 8 connection(s)
I20260929 16:41:53.431970 5427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:41:53.432519 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 5427
I20260929 16:41:53.432713 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/master-0/wal/instance
I20260929 16:41:53.433574 789 external_mini_cluster.cc:1428] Setting key 26490dcbcea852a94d17565874757f46
I20260929 16:41:53.436828 5493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:41:53.444159 5496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:53.435961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52830 (local address 127.0.197.126:42879)
0929 16:41:53.437090 (+ 1129us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:53.437121 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:53.437182 (+ 61us) server_negotiation.cc:407] Connection header received
0929 16:41:53.439535 (+ 2353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:53.439565 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:53.440198 (+ 633us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:53.440701 (+ 503us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:53.441806 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:53.441816 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:53.441839 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:41:53.441867 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:53.441963 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:41:53.441985 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:53.442246 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:53.442384 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 16:41:53.442713 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":349,"threads_started":1}
I20260929 16:41:53.450974 5493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e99dcbdc84474c229dfaf49a99ade2dc: Bootstrap starting.
I20260929 16:41:53.454886 5493 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e99dcbdc84474c229dfaf49a99ade2dc: Neither blocks nor log segments found. Creating new log.
I20260929 16:41:53.456156 5493 log.cc:824] T 00000000000000000000000000000000 P e99dcbdc84474c229dfaf49a99ade2dc: Log is configured to *not* fsync() on all Append() calls
I20260929 16:41:53.459702 5493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e99dcbdc84474c229dfaf49a99ade2dc: No bootstrap required, opened a new log
I20260929 16:41:53.469311 5493 raft_consensus.cc:348] T 00000000000000000000000000000000 P e99dcbdc84474c229dfaf49a99ade2dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e99dcbdc84474c229dfaf49a99ade2dc" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 42879 } }
I20260929 16:41:53.469786 5493 raft_consensus.cc:374] T 00000000000000000000000000000000 P e99dcbdc84474c229dfaf49a99ade2dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:41:53.469913 5493 raft_consensus.cc:729] T 00000000000000000000000000000000 P e99dcbdc84474c229dfaf49a99ade2dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e99dcbdc84474c229dfaf49a99ade2dc, State: Initialized, Role: FOLLOWER
I20260929 16:41:53.470541 5493 consensus_queue.cc:260] T 00000000000000000000000000000000 P e99dcbdc84474c229dfaf49a99ade2dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e99dcbdc84474c229dfaf49a99ade2dc" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 42879 } }
I20260929 16:41:53.470801 5493 raft_consensus.cc:388] T 00000000000000000000000000000000 P e99dcbdc84474c229dfaf49a99ade2dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:41:53.470970 5493 raft_consensus.cc:482] T 00000000000000000000000000000000 P e99dcbdc84474c229dfaf49a99ade2dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:41:53.471226 5493 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e99dcbdc84474c229dfaf49a99ade2dc [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:41:53.473544 5493 raft_consensus.cc:504] T 00000000000000000000000000000000 P e99dcbdc84474c229dfaf49a99ade2dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e99dcbdc84474c229dfaf49a99ade2dc" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 42879 } }
I20260929 16:41:53.474179 5493 leader_election.cc:302] T 00000000000000000000000000000000 P e99dcbdc84474c229dfaf49a99ade2dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e99dcbdc84474c229dfaf49a99ade2dc; no voters:
I20260929 16:41:53.474939 5493 leader_election.cc:288] T 00000000000000000000000000000000 P e99dcbdc84474c229dfaf49a99ade2dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:41:53.475219 5498 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e99dcbdc84474c229dfaf49a99ade2dc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:41:53.476693 5498 raft_consensus.cc:686] T 00000000000000000000000000000000 P e99dcbdc84474c229dfaf49a99ade2dc [term 1 LEADER]: Becoming Leader. State: Replica: e99dcbdc84474c229dfaf49a99ade2dc, State: Running, Role: LEADER
I20260929 16:41:53.477469 5498 consensus_queue.cc:237] T 00000000000000000000000000000000 P e99dcbdc84474c229dfaf49a99ade2dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e99dcbdc84474c229dfaf49a99ade2dc" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 42879 } }
I20260929 16:41:53.478077 5493 sys_catalog.cc:564] T 00000000000000000000000000000000 P e99dcbdc84474c229dfaf49a99ade2dc [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:41:53.481107 5499 sys_catalog.cc:455] T 00000000000000000000000000000000 P e99dcbdc84474c229dfaf49a99ade2dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e99dcbdc84474c229dfaf49a99ade2dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e99dcbdc84474c229dfaf49a99ade2dc" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 42879 } } }
I20260929 16:41:53.481140 5500 sys_catalog.cc:455] T 00000000000000000000000000000000 P e99dcbdc84474c229dfaf49a99ade2dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader e99dcbdc84474c229dfaf49a99ade2dc. Latest consensus state: current_term: 1 leader_uuid: "e99dcbdc84474c229dfaf49a99ade2dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e99dcbdc84474c229dfaf49a99ade2dc" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 42879 } } }
I20260929 16:41:53.481595 5499 sys_catalog.cc:458] T 00000000000000000000000000000000 P e99dcbdc84474c229dfaf49a99ade2dc [sys.catalog]: This master's current role is: LEADER
I20260929 16:41:53.481624 5500 sys_catalog.cc:458] T 00000000000000000000000000000000 P e99dcbdc84474c229dfaf49a99ade2dc [sys.catalog]: This master's current role is: LEADER
I20260929 16:41:53.484211 5505 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:41:53.490985 5505 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:41:53.502000 5505 catalog_manager.cc:1347] Generated new cluster ID: fc4321bf32b9474791e6faa30e2f51ec
I20260929 16:41:53.502159 5505 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:41:53.522102 5505 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:41:53.522665 5505 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:41:53.532631 5505 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e99dcbdc84474c229dfaf49a99ade2dc: Generated new TSK 0
I20260929 16:41:53.533854 5505 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:41:53.548348 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.65:0
--local_ip_for_outbound_sockets=127.0.197.65
--webserver_interface=127.0.197.65
--webserver_port=0
--tserver_master_addrs=127.0.197.126:42879
--builtin_ntp_servers=127.0.197.84:34767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 16:41:53.906196 5517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:41:53.906546 5517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:41:53.906692 5517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:41:53.914592 5517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:41:53.914716 5517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:41:53.914911 5517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.65
I20260929 16:41:53.924502 5517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:34767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.197.65
--webserver_port=0
--tserver_master_addrs=127.0.197.126:42879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.65
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:41:53.925930 5517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:41:53.927906 5517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:41:53.942448 5522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:53.942543 5523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:53.942515 5525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:41:53.943881 5517 server_base.cc:1034] running on GCE node
I20260929 16:41:53.944531 5517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:41:53.945626 5517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:41:53.946846 5517 hybrid_clock.cc:648] HybridClock initialized: now 1790700113946798 us; error 36 us; skew 500 ppm
I20260929 16:41:53.947254 5517 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:41:53.949772 5517 webserver.cc:460] Webserver started at http://127.0.197.65:41869/ using document root <none> and password file <none>
I20260929 16:41:53.950594 5517 fs_manager.cc:359] Metadata directory not provided
I20260929 16:41:53.950688 5517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:41:53.951141 5517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:41:53.954411 5517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/ts-0/data/instance:
uuid: "f3f4779c99f4471f9665f121adb398dc"
format_stamp: "Formatted at 2026-09-29 16:41:53 on dist-test-slave-l31f"
server_key: "e7f55cdb7c612d6462b95308f08a4684"
server_key_iv: "d0b944234537f6bec602f4c75ad50628"
server_key_version: "encryptionkey@0"
I20260929 16:41:53.955473 5517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/ts-0/wal/instance:
uuid: "f3f4779c99f4471f9665f121adb398dc"
format_stamp: "Formatted at 2026-09-29 16:41:53 on dist-test-slave-l31f"
server_key: "e7f55cdb7c612d6462b95308f08a4684"
server_key_iv: "d0b944234537f6bec602f4c75ad50628"
server_key_version: "encryptionkey@0"
I20260929 16:41:53.961239 5517 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260929 16:41:53.965224 5531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:53.966691 5517 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 16:41:53.966907 5517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/ts-0/wal
uuid: "f3f4779c99f4471f9665f121adb398dc"
format_stamp: "Formatted at 2026-09-29 16:41:53 on dist-test-slave-l31f"
server_key: "e7f55cdb7c612d6462b95308f08a4684"
server_key_iv: "d0b944234537f6bec602f4c75ad50628"
server_key_version: "encryptionkey@0"
I20260929 16:41:53.967250 5517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:41:53.973567 5517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:41:53.974515 5517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:41:53.974833 5517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:41:53.976065 5517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:41:53.977934 5517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:41:53.978089 5517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:53.978248 5517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:41:53.978348 5517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:54.014389 5517 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.65:40663
I20260929 16:41:54.014433 5643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.65:40663 every 8 connection(s)
I20260929 16:41:54.015918 5517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
I20260929 16:41:54.022207 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 5517
I20260929 16:41:54.022428 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/ts-0/wal/instance
I20260929 16:41:54.023267 789 external_mini_cluster.cc:1428] Setting key cddf76f1564b074e48937922daa06cae
I20260929 16:41:54.025496 5646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:54.018874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:60743 (local address 127.0.197.126:42879)
0929 16:41:54.019621 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:54.019627 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:54.020815 (+ 1188us) server_negotiation.cc:407] Connection header received
0929 16:41:54.022723 (+ 1908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:54.022731 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:54.022810 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:54.022955 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:54.024580 (+ 1625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:54.024584 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:54.024587 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:41:54.024605 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:54.024651 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:41:54.024663 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:54.024774 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:54.025060 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 16:41:54.025159 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":353,"threads_started":1}
I20260929 16:41:54.026396 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.66:0
--local_ip_for_outbound_sockets=127.0.197.66
--webserver_interface=127.0.197.66
--webserver_port=0
--tserver_master_addrs=127.0.197.126:42879
--builtin_ntp_servers=127.0.197.84:34767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 16:41:54.026582 5647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:54.019605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:42879 (local address 127.0.197.65:60743)
0929 16:41:54.020622 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:54.020673 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:54.022285 (+ 1612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:54.023260 (+ 975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:54.023270 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:54.023408 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:41:54.024209 (+ 801us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:41:54.024212 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:41:54.024255 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:41:54.024388 (+ 133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:41:54.024782 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:41:54.024793 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:41:54.024802 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 16:41:54.025118 (+ 316us) client_negotiation.cc:240] Negotiation successful
0929 16:41:54.025454 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":296,"threads_started":1}
I20260929 16:41:54.028319 5644 heartbeater.cc:347] Connected to a master server at 127.0.197.126:42879
I20260929 16:41:54.028661 5644 heartbeater.cc:464] Registering TS with master...
I20260929 16:41:54.029273 5644 heartbeater.cc:511] Master 127.0.197.126:42879 requested a full tablet report, sending...
I20260929 16:41:54.031631 5458 ts_manager.cc:194] Registered new tserver with Master: f3f4779c99f4471f9665f121adb398dc (127.0.197.65:40663)
W20260929 16:41:54.372201 5648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:41:54.372524 5648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:41:54.372684 5648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:41:54.380388 5648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:41:54.380507 5648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:41:54.380702 5648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.66
I20260929 16:41:54.390285 5648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:34767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.197.66
--webserver_port=0
--tserver_master_addrs=127.0.197.126:42879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.66
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:41:54.391757 5648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:41:54.393656 5648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:41:54.408506 5653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:54.408561 5656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:54.408593 5654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:41:54.410280 5648 server_base.cc:1034] running on GCE node
I20260929 16:41:54.411118 5648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:41:54.412171 5648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:41:54.413424 5648 hybrid_clock.cc:648] HybridClock initialized: now 1790700114413367 us; error 35 us; skew 500 ppm
I20260929 16:41:54.413837 5648 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:41:54.416502 5648 webserver.cc:460] Webserver started at http://127.0.197.66:35153/ using document root <none> and password file <none>
I20260929 16:41:54.417339 5648 fs_manager.cc:359] Metadata directory not provided
I20260929 16:41:54.417462 5648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:41:54.417820 5648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:41:54.421234 5648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/ts-1/data/instance:
uuid: "c9688a355aa647069e7218293e5134c0"
format_stamp: "Formatted at 2026-09-29 16:41:54 on dist-test-slave-l31f"
server_key: "61827dadae939a879ca95bf94fe6c097"
server_key_iv: "d07137ab12e26474a96ce8ccc7c87dbd"
server_key_version: "encryptionkey@0"
I20260929 16:41:54.422236 5648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/ts-1/wal/instance:
uuid: "c9688a355aa647069e7218293e5134c0"
format_stamp: "Formatted at 2026-09-29 16:41:54 on dist-test-slave-l31f"
server_key: "61827dadae939a879ca95bf94fe6c097"
server_key_iv: "d07137ab12e26474a96ce8ccc7c87dbd"
server_key_version: "encryptionkey@0"
I20260929 16:41:54.428287 5648 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260929 16:41:54.432426 5662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:54.434084 5648 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:41:54.434283 5648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/ts-1/wal
uuid: "c9688a355aa647069e7218293e5134c0"
format_stamp: "Formatted at 2026-09-29 16:41:54 on dist-test-slave-l31f"
server_key: "61827dadae939a879ca95bf94fe6c097"
server_key_iv: "d07137ab12e26474a96ce8ccc7c87dbd"
server_key_version: "encryptionkey@0"
I20260929 16:41:54.434561 5648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:41:54.441059 5648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:41:54.441970 5648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:41:54.442257 5648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:41:54.443405 5648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:41:54.444993 5648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:41:54.445091 5648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:54.445228 5648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:41:54.445304 5648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:54.483075 5648 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.66:40259
I20260929 16:41:54.483178 5774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.66:40259 every 8 connection(s)
I20260929 16:41:54.484539 5648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
I20260929 16:41:54.488574 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 5648
I20260929 16:41:54.488793 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/ts-1/wal/instance
I20260929 16:41:54.489369 789 external_mini_cluster.cc:1428] Setting key 4ba8578784b9b0adb68371d365cceabd
I20260929 16:41:54.492709 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.67:0
--local_ip_for_outbound_sockets=127.0.197.67
--webserver_interface=127.0.197.67
--webserver_port=0
--tserver_master_addrs=127.0.197.126:42879
--builtin_ntp_servers=127.0.197.84:34767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 16:41:54.494972 5646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:54.487672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:38229 (local address 127.0.197.126:42879)
0929 16:41:54.487910 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:54.487915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:54.489723 (+ 1808us) server_negotiation.cc:407] Connection header received
0929 16:41:54.492074 (+ 2351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:54.492080 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:54.492167 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:54.492284 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:54.493806 (+ 1522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:54.493815 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:54.493822 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:41:54.493851 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:54.493901 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:41:54.493920 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:54.494063 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:54.494532 (+ 469us) server_negotiation.cc:299] Negotiation successful
0929 16:41:54.494607 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260929 16:41:54.496253 5777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:54.488223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:42879 (local address 127.0.197.66:38229)
0929 16:41:54.489504 (+ 1281us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:54.489565 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:54.491748 (+ 2183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:54.492543 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:54.492554 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:54.492661 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:41:54.493454 (+ 793us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:41:54.493456 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:41:54.493499 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:41:54.493598 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:41:54.494144 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:41:54.494162 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:41:54.494173 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 16:41:54.494607 (+ 434us) client_negotiation.cc:240] Negotiation successful
0929 16:41:54.494945 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":362,"threads_started":1}
I20260929 16:41:54.497920 5775 heartbeater.cc:347] Connected to a master server at 127.0.197.126:42879
I20260929 16:41:54.498294 5775 heartbeater.cc:464] Registering TS with master...
I20260929 16:41:54.498968 5775 heartbeater.cc:511] Master 127.0.197.126:42879 requested a full tablet report, sending...
I20260929 16:41:54.500789 5458 ts_manager.cc:194] Registered new tserver with Master: c9688a355aa647069e7218293e5134c0 (127.0.197.66:40259)
W20260929 16:41:54.856256 5778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:41:54.856575 5778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:41:54.856673 5778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:41:54.864362 5778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:41:54.864485 5778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:41:54.864598 5778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.67
I20260929 16:41:54.874207 5778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:34767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.197.67
--webserver_port=0
--tserver_master_addrs=127.0.197.126:42879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.67
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:41:54.875597 5778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:41:54.877410 5778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:41:54.892561 5783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:54.892663 5784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:54.892628 5786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:41:54.893528 5778 server_base.cc:1034] running on GCE node
I20260929 16:41:54.894215 5778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:41:54.895383 5778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:41:54.896740 5778 hybrid_clock.cc:648] HybridClock initialized: now 1790700114896646 us; error 73 us; skew 500 ppm
I20260929 16:41:54.897141 5778 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:41:54.900074 5778 webserver.cc:460] Webserver started at http://127.0.197.67:34793/ using document root <none> and password file <none>
I20260929 16:41:54.900967 5778 fs_manager.cc:359] Metadata directory not provided
I20260929 16:41:54.901067 5778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:41:54.901355 5778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:41:54.904580 5778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/ts-2/data/instance:
uuid: "0684b0d755ae4f238fbb08fd90659745"
format_stamp: "Formatted at 2026-09-29 16:41:54 on dist-test-slave-l31f"
server_key: "3b703a78e7007a3084e96a6da7a72abb"
server_key_iv: "2653466dced370355f6a8013192f03b6"
server_key_version: "encryptionkey@0"
I20260929 16:41:54.905488 5778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/ts-2/wal/instance:
uuid: "0684b0d755ae4f238fbb08fd90659745"
format_stamp: "Formatted at 2026-09-29 16:41:54 on dist-test-slave-l31f"
server_key: "3b703a78e7007a3084e96a6da7a72abb"
server_key_iv: "2653466dced370355f6a8013192f03b6"
server_key_version: "encryptionkey@0"
I20260929 16:41:54.911679 5778 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260929 16:41:54.916522 5792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:54.918313 5778 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:41:54.918574 5778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/ts-2/wal
uuid: "0684b0d755ae4f238fbb08fd90659745"
format_stamp: "Formatted at 2026-09-29 16:41:54 on dist-test-slave-l31f"
server_key: "3b703a78e7007a3084e96a6da7a72abb"
server_key_iv: "2653466dced370355f6a8013192f03b6"
server_key_version: "encryptionkey@0"
I20260929 16:41:54.918869 5778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:41:54.925729 5778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:41:54.926693 5778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:41:54.927055 5778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:41:54.928153 5778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:41:54.929770 5778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:41:54.929903 5778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:54.930056 5778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:41:54.930145 5778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:54.968569 5778 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.67:44363
I20260929 16:41:54.968628 5904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.67:44363 every 8 connection(s)
I20260929 16:41:54.970042 5778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
I20260929 16:41:54.977025 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 5778
I20260929 16:41:54.977201 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/ts-2/wal/instance
I20260929 16:41:54.977689 789 external_mini_cluster.cc:1428] Setting key 115a1052cd2a501aaec340478d8d0091
I20260929 16:41:54.981077 5646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:54.973197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:50917 (local address 127.0.197.126:42879)
0929 16:41:54.973392 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:54.973397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:54.975637 (+ 2240us) server_negotiation.cc:407] Connection header received
0929 16:41:54.978104 (+ 2467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:54.978111 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:54.978210 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:54.978335 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:54.980244 (+ 1909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:54.980257 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:54.980267 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:41:54.980286 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:54.980320 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:41:54.980332 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:54.980470 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:54.980703 (+ 233us) server_negotiation.cc:299] Negotiation successful
0929 16:41:54.980776 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 16:41:54.982357 5907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:54.973768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:42879 (local address 127.0.197.67:50917)
0929 16:41:54.975370 (+ 1602us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:54.975435 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:54.977645 (+ 2210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:54.978632 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:54.978643 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:54.978782 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:41:54.979867 (+ 1085us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:41:54.979870 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:41:54.979916 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:41:54.980023 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:41:54.980440 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:41:54.980451 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:41:54.980460 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 16:41:54.980806 (+ 346us) client_negotiation.cc:240] Negotiation successful
0929 16:41:54.981144 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"thread_start_us":428,"threads_started":1}
I20260929 16:41:54.984016 5905 heartbeater.cc:347] Connected to a master server at 127.0.197.126:42879
I20260929 16:41:54.984323 5905 heartbeater.cc:464] Registering TS with master...
I20260929 16:41:54.985031 5905 heartbeater.cc:511] Master 127.0.197.126:42879 requested a full tablet report, sending...
I20260929 16:41:54.987095 5458 ts_manager.cc:194] Registered new tserver with Master: 0684b0d755ae4f238fbb08fd90659745 (127.0.197.67:44363)
I20260929 16:41:54.996153 789 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:41:55.007938 5646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:55.004004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34836 (local address 127.0.197.126:42879)
0929 16:41:55.004231 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:55.004239 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:55.005425 (+ 1186us) server_negotiation.cc:407] Connection header received
0929 16:41:55.005518 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:55.005524 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:55.005621 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:55.005756 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:55.007250 (+ 1494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:55.007256 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:55.007261 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:41:55.007292 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:55.007330 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:41:55.007344 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:55.007492 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:55.007572 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 16:41:55.007639 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 16:41:55.012914 5458 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34836:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:41:55.019971 5458 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:41:55.035094 5644 heartbeater.cc:503] Master 127.0.197.126:42879 was elected leader, sending a full tablet report...
I20260929 16:41:55.050797 5917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:55.033767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:40663 (local address 127.0.0.1:58236)
0929 16:41:55.046330 (+ 12563us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:55.046352 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:55.046480 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:55.047268 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:55.047277 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:55.047395 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:41:55.048334 (+ 939us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:41:55.048340 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:41:55.048376 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:41:55.048473 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:41:55.049904 (+ 1431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:41:55.049916 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:41:55.049924 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 16:41:55.050085 (+ 161us) client_negotiation.cc:240] Negotiation successful
0929 16:41:55.050188 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12397,"thread_start_us":15052,"threads_started":1}
I20260929 16:41:55.052058 5919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:55.034036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58236 (local address 127.0.197.65:40663)
0929 16:41:55.045718 (+ 11682us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:55.045730 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:55.046608 (+ 878us) server_negotiation.cc:407] Connection header received
0929 16:41:55.046688 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:55.046696 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:55.046991 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:55.047990 (+ 999us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:55.049652 (+ 1662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:55.049661 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:55.049675 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:41:55.049692 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:55.049746 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:41:55.049766 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:55.051477 (+ 1711us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:55.051558 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 16:41:55.051682 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11485,"thread_start_us":11320,"threads_started":1}
I20260929 16:41:55.053745 5918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:55.036304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.67:44363 (local address 127.0.0.1:50326)
0929 16:41:55.049134 (+ 12830us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:55.049157 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:55.049362 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:55.050555 (+ 1193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:55.050563 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:55.050607 (+ 44us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:41:55.052192 (+ 1585us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:41:55.052195 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:41:55.052216 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:41:55.052267 (+ 51us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:41:55.053067 (+ 800us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:41:55.053072 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:41:55.053077 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 16:41:55.053221 (+ 144us) client_negotiation.cc:240] Negotiation successful
0929 16:41:55.053285 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12665,"thread_start_us":6197,"threads_started":1}
I20260929 16:41:55.054271 5920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:55.034024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50494 (local address 127.0.197.66:40259)
0929 16:41:55.051214 (+ 17190us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:55.051224 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:55.051255 (+ 31us) server_negotiation.cc:407] Connection header received
0929 16:41:55.051323 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:55.051331 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:55.051532 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:55.051720 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:55.053102 (+ 1382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:55.053109 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:55.053121 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:41:55.053145 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:55.053191 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:41:55.053210 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:55.053345 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:55.053848 (+ 503us) server_negotiation.cc:299] Negotiation successful
0929 16:41:55.053949 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17017,"thread_start_us":17106,"threads_started":1}
I20260929 16:41:55.054376 5921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:55.035543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50326 (local address 127.0.197.67:44363)
0929 16:41:55.048648 (+ 13105us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:55.048659 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:55.049268 (+ 609us) server_negotiation.cc:407] Connection header received
0929 16:41:55.049525 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:55.049535 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:55.049797 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:55.049941 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:55.052699 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:55.052709 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:55.052724 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:41:55.052744 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:55.052817 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:41:55.052839 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:55.053008 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:55.053678 (+ 670us) server_negotiation.cc:299] Negotiation successful
0929 16:41:55.053790 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12909,"thread_start_us":12740,"threads_started":1}
I20260929 16:41:55.055475 5922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:55.033628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.66:40259 (local address 127.0.0.1:50494)
0929 16:41:55.043164 (+ 9536us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:55.043214 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:55.043441 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:55.051740 (+ 8299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:55.051746 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:55.051771 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:41:55.052473 (+ 702us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:41:55.052476 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:41:55.052491 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:41:55.052528 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:41:55.053398 (+ 870us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:41:55.053402 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:41:55.053412 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 16:41:55.053641 (+ 229us) client_negotiation.cc:240] Negotiation successful
0929 16:41:55.053700 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9079,"thread_start_us":12369,"threads_started":1}
I20260929 16:41:55.056509 5579 tablet_service.cc:1467] Processing CreateTablet for tablet 2af6372bb77e4533a38bc100eaba12e2 (DEFAULT_TABLE table=test-table [id=4ea8991b1c2f49868a031f8075ac7962]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:41:55.058101 5579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2af6372bb77e4533a38bc100eaba12e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:41:55.057988 5840 tablet_service.cc:1467] Processing CreateTablet for tablet 2af6372bb77e4533a38bc100eaba12e2 (DEFAULT_TABLE table=test-table [id=4ea8991b1c2f49868a031f8075ac7962]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:41:55.059401 5840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2af6372bb77e4533a38bc100eaba12e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:41:55.059247 5710 tablet_service.cc:1467] Processing CreateTablet for tablet 2af6372bb77e4533a38bc100eaba12e2 (DEFAULT_TABLE table=test-table [id=4ea8991b1c2f49868a031f8075ac7962]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:41:55.060658 5710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2af6372bb77e4533a38bc100eaba12e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:41:55.071849 5923 tablet_bootstrap.cc:492] T 2af6372bb77e4533a38bc100eaba12e2 P 0684b0d755ae4f238fbb08fd90659745: Bootstrap starting.
I20260929 16:41:55.074404 5924 tablet_bootstrap.cc:492] T 2af6372bb77e4533a38bc100eaba12e2 P f3f4779c99f4471f9665f121adb398dc: Bootstrap starting.
I20260929 16:41:55.076208 5925 tablet_bootstrap.cc:492] T 2af6372bb77e4533a38bc100eaba12e2 P c9688a355aa647069e7218293e5134c0: Bootstrap starting.
I20260929 16:41:55.077579 5923 tablet_bootstrap.cc:654] T 2af6372bb77e4533a38bc100eaba12e2 P 0684b0d755ae4f238fbb08fd90659745: Neither blocks nor log segments found. Creating new log.
I20260929 16:41:55.078275 5924 tablet_bootstrap.cc:654] T 2af6372bb77e4533a38bc100eaba12e2 P f3f4779c99f4471f9665f121adb398dc: Neither blocks nor log segments found. Creating new log.
I20260929 16:41:55.079042 5923 log.cc:824] T 2af6372bb77e4533a38bc100eaba12e2 P 0684b0d755ae4f238fbb08fd90659745: Log is configured to *not* fsync() on all Append() calls
I20260929 16:41:55.079388 5925 tablet_bootstrap.cc:654] T 2af6372bb77e4533a38bc100eaba12e2 P c9688a355aa647069e7218293e5134c0: Neither blocks nor log segments found. Creating new log.
I20260929 16:41:55.079576 5924 log.cc:824] T 2af6372bb77e4533a38bc100eaba12e2 P f3f4779c99f4471f9665f121adb398dc: Log is configured to *not* fsync() on all Append() calls
I20260929 16:41:55.080675 5925 log.cc:824] T 2af6372bb77e4533a38bc100eaba12e2 P c9688a355aa647069e7218293e5134c0: Log is configured to *not* fsync() on all Append() calls
I20260929 16:41:55.083691 5923 tablet_bootstrap.cc:492] T 2af6372bb77e4533a38bc100eaba12e2 P 0684b0d755ae4f238fbb08fd90659745: No bootstrap required, opened a new log
I20260929 16:41:55.084105 5923 ts_tablet_manager.cc:1397] T 2af6372bb77e4533a38bc100eaba12e2 P 0684b0d755ae4f238fbb08fd90659745: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260929 16:41:55.086845 5924 tablet_bootstrap.cc:492] T 2af6372bb77e4533a38bc100eaba12e2 P f3f4779c99f4471f9665f121adb398dc: No bootstrap required, opened a new log
I20260929 16:41:55.086886 5925 tablet_bootstrap.cc:492] T 2af6372bb77e4533a38bc100eaba12e2 P c9688a355aa647069e7218293e5134c0: No bootstrap required, opened a new log
I20260929 16:41:55.087194 5924 ts_tablet_manager.cc:1397] T 2af6372bb77e4533a38bc100eaba12e2 P f3f4779c99f4471f9665f121adb398dc: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.003s
I20260929 16:41:55.087196 5925 ts_tablet_manager.cc:1397] T 2af6372bb77e4533a38bc100eaba12e2 P c9688a355aa647069e7218293e5134c0: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260929 16:41:55.094920 5924 raft_consensus.cc:348] T 2af6372bb77e4533a38bc100eaba12e2 P f3f4779c99f4471f9665f121adb398dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9688a355aa647069e7218293e5134c0" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 40259 } } peers { permanent_uuid: "f3f4779c99f4471f9665f121adb398dc" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 40663 } } peers { permanent_uuid: "0684b0d755ae4f238fbb08fd90659745" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 44363 } }
I20260929 16:41:55.095474 5924 raft_consensus.cc:374] T 2af6372bb77e4533a38bc100eaba12e2 P f3f4779c99f4471f9665f121adb398dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:41:55.095573 5924 raft_consensus.cc:729] T 2af6372bb77e4533a38bc100eaba12e2 P f3f4779c99f4471f9665f121adb398dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3f4779c99f4471f9665f121adb398dc, State: Initialized, Role: FOLLOWER
I20260929 16:41:55.095754 5923 raft_consensus.cc:348] T 2af6372bb77e4533a38bc100eaba12e2 P 0684b0d755ae4f238fbb08fd90659745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9688a355aa647069e7218293e5134c0" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 40259 } } peers { permanent_uuid: "f3f4779c99f4471f9665f121adb398dc" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 40663 } } peers { permanent_uuid: "0684b0d755ae4f238fbb08fd90659745" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 44363 } }
I20260929 16:41:55.096262 5923 raft_consensus.cc:374] T 2af6372bb77e4533a38bc100eaba12e2 P 0684b0d755ae4f238fbb08fd90659745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:41:55.096379 5923 raft_consensus.cc:729] T 2af6372bb77e4533a38bc100eaba12e2 P 0684b0d755ae4f238fbb08fd90659745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0684b0d755ae4f238fbb08fd90659745, State: Initialized, Role: FOLLOWER
I20260929 16:41:55.096320 5924 consensus_queue.cc:260] T 2af6372bb77e4533a38bc100eaba12e2 P f3f4779c99f4471f9665f121adb398dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9688a355aa647069e7218293e5134c0" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 40259 } } peers { permanent_uuid: "f3f4779c99f4471f9665f121adb398dc" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 40663 } } peers { permanent_uuid: "0684b0d755ae4f238fbb08fd90659745" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 44363 } }
I20260929 16:41:55.097113 5923 consensus_queue.cc:260] T 2af6372bb77e4533a38bc100eaba12e2 P 0684b0d755ae4f238fbb08fd90659745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9688a355aa647069e7218293e5134c0" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 40259 } } peers { permanent_uuid: "f3f4779c99f4471f9665f121adb398dc" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 40663 } } peers { permanent_uuid: "0684b0d755ae4f238fbb08fd90659745" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 44363 } }
I20260929 16:41:55.098497 5924 ts_tablet_manager.cc:1428] T 2af6372bb77e4533a38bc100eaba12e2 P f3f4779c99f4471f9665f121adb398dc: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260929 16:41:55.099668 5905 heartbeater.cc:503] Master 127.0.197.126:42879 was elected leader, sending a full tablet report...
I20260929 16:41:55.100294 5923 ts_tablet_manager.cc:1428] T 2af6372bb77e4533a38bc100eaba12e2 P 0684b0d755ae4f238fbb08fd90659745: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260929 16:41:55.099112 5925 raft_consensus.cc:348] T 2af6372bb77e4533a38bc100eaba12e2 P c9688a355aa647069e7218293e5134c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9688a355aa647069e7218293e5134c0" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 40259 } } peers { permanent_uuid: "f3f4779c99f4471f9665f121adb398dc" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 40663 } } peers { permanent_uuid: "0684b0d755ae4f238fbb08fd90659745" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 44363 } }
I20260929 16:41:55.100819 5925 raft_consensus.cc:374] T 2af6372bb77e4533a38bc100eaba12e2 P c9688a355aa647069e7218293e5134c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:41:55.100945 5925 raft_consensus.cc:729] T 2af6372bb77e4533a38bc100eaba12e2 P c9688a355aa647069e7218293e5134c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9688a355aa647069e7218293e5134c0, State: Initialized, Role: FOLLOWER
I20260929 16:41:55.101616 5925 consensus_queue.cc:260] T 2af6372bb77e4533a38bc100eaba12e2 P c9688a355aa647069e7218293e5134c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9688a355aa647069e7218293e5134c0" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 40259 } } peers { permanent_uuid: "f3f4779c99f4471f9665f121adb398dc" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 40663 } } peers { permanent_uuid: "0684b0d755ae4f238fbb08fd90659745" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 44363 } }
I20260929 16:41:55.103117 5775 heartbeater.cc:503] Master 127.0.197.126:42879 was elected leader, sending a full tablet report...
I20260929 16:41:55.103871 5925 ts_tablet_manager.cc:1428] T 2af6372bb77e4533a38bc100eaba12e2 P c9688a355aa647069e7218293e5134c0: Time spent starting tablet: real 0.017s user 0.011s sys 0.004s
I20260929 16:41:55.129563 5930 raft_consensus.cc:482] T 2af6372bb77e4533a38bc100eaba12e2 P 0684b0d755ae4f238fbb08fd90659745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:41:55.129944 5930 raft_consensus.cc:504] T 2af6372bb77e4533a38bc100eaba12e2 P 0684b0d755ae4f238fbb08fd90659745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9688a355aa647069e7218293e5134c0" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 40259 } } peers { permanent_uuid: "f3f4779c99f4471f9665f121adb398dc" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 40663 } } peers { permanent_uuid: "0684b0d755ae4f238fbb08fd90659745" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 44363 } }
I20260929 16:41:55.132009 5930 leader_election.cc:288] T 2af6372bb77e4533a38bc100eaba12e2 P 0684b0d755ae4f238fbb08fd90659745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9688a355aa647069e7218293e5134c0 (127.0.197.66:40259), f3f4779c99f4471f9665f121adb398dc (127.0.197.65:40663)
I20260929 16:41:55.134920 5920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:55.132600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:46253 (local address 127.0.197.66:40259)
0929 16:41:55.132786 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:55.132792 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:55.132811 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:41:55.132951 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:55.132962 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:55.133010 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:55.133133 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:55.134301 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:55.134307 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:55.134313 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:41:55.134338 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:55.134369 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:41:55.134381 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:55.134511 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:55.134592 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 16:41:55.134653 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 16:41:55.134929 5907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:55.132369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.66:40259 (local address 127.0.197.67:46253)
0929 16:41:55.132657 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:55.132679 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:55.132790 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:55.133179 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:55.133185 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:55.133224 (+ 39us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:41:55.134049 (+ 825us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:41:55.134052 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:41:55.134071 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:41:55.134132 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:41:55.134484 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:41:55.134488 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:41:55.134492 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 16:41:55.134598 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 16:41:55.134653 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153}
I20260929 16:41:55.136157 5730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2af6372bb77e4533a38bc100eaba12e2" candidate_uuid: "0684b0d755ae4f238fbb08fd90659745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9688a355aa647069e7218293e5134c0" is_pre_election: true
I20260929 16:41:55.136615 5919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:55.132563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:37661 (local address 127.0.197.65:40663)
0929 16:41:55.132748 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:55.132760 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:55.134294 (+ 1534us) server_negotiation.cc:407] Connection header received
0929 16:41:55.134547 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:55.134553 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:55.134619 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:55.134752 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:55.135981 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:55.135987 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:55.135992 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:41:55.136019 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:55.136052 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:41:55.136064 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:55.136206 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:55.136264 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 16:41:55.136335 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 16:41:55.136767 5932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:55.132992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:40663 (local address 127.0.197.67:37661)
0929 16:41:55.134168 (+ 1176us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:55.134183 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:55.134323 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:55.134786 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:55.134792 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:55.134888 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:41:55.135757 (+ 869us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:41:55.135760 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:41:55.135774 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:41:55.135812 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:41:55.136162 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:41:55.136166 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:41:55.136171 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 16:41:55.136273 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 16:41:55.136324 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1069,"thread_start_us":446,"threads_started":1}
I20260929 16:41:55.136758 5730 raft_consensus.cc:2454] T 2af6372bb77e4533a38bc100eaba12e2 P c9688a355aa647069e7218293e5134c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0684b0d755ae4f238fbb08fd90659745 in term 0.
I20260929 16:41:55.137477 5793 leader_election.cc:302] T 2af6372bb77e4533a38bc100eaba12e2 P 0684b0d755ae4f238fbb08fd90659745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0684b0d755ae4f238fbb08fd90659745, c9688a355aa647069e7218293e5134c0; no voters:
I20260929 16:41:55.137738 5599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2af6372bb77e4533a38bc100eaba12e2" candidate_uuid: "0684b0d755ae4f238fbb08fd90659745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3f4779c99f4471f9665f121adb398dc" is_pre_election: true
I20260929 16:41:55.137948 5930 raft_consensus.cc:2781] T 2af6372bb77e4533a38bc100eaba12e2 P 0684b0d755ae4f238fbb08fd90659745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:41:55.138108 5930 raft_consensus.cc:482] T 2af6372bb77e4533a38bc100eaba12e2 P 0684b0d755ae4f238fbb08fd90659745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:41:55.138224 5930 raft_consensus.cc:3037] T 2af6372bb77e4533a38bc100eaba12e2 P 0684b0d755ae4f238fbb08fd90659745 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:41:55.138428 5599 raft_consensus.cc:2454] T 2af6372bb77e4533a38bc100eaba12e2 P f3f4779c99f4471f9665f121adb398dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0684b0d755ae4f238fbb08fd90659745 in term 0.
I20260929 16:41:55.141062 5930 raft_consensus.cc:504] T 2af6372bb77e4533a38bc100eaba12e2 P 0684b0d755ae4f238fbb08fd90659745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9688a355aa647069e7218293e5134c0" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 40259 } } peers { permanent_uuid: "f3f4779c99f4471f9665f121adb398dc" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 40663 } } peers { permanent_uuid: "0684b0d755ae4f238fbb08fd90659745" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 44363 } }
I20260929 16:41:55.141830 5930 leader_election.cc:288] T 2af6372bb77e4533a38bc100eaba12e2 P 0684b0d755ae4f238fbb08fd90659745 [CANDIDATE]: Term 1 election: Requested vote from peers c9688a355aa647069e7218293e5134c0 (127.0.197.66:40259), f3f4779c99f4471f9665f121adb398dc (127.0.197.65:40663)
I20260929 16:41:55.142520 5730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2af6372bb77e4533a38bc100eaba12e2" candidate_uuid: "0684b0d755ae4f238fbb08fd90659745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9688a355aa647069e7218293e5134c0"
I20260929 16:41:55.142756 5730 raft_consensus.cc:3037] T 2af6372bb77e4533a38bc100eaba12e2 P c9688a355aa647069e7218293e5134c0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:41:55.143224 5599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2af6372bb77e4533a38bc100eaba12e2" candidate_uuid: "0684b0d755ae4f238fbb08fd90659745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3f4779c99f4471f9665f121adb398dc"
I20260929 16:41:55.143711 5599 raft_consensus.cc:3037] T 2af6372bb77e4533a38bc100eaba12e2 P f3f4779c99f4471f9665f121adb398dc [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:41:55.145421 5730 raft_consensus.cc:2454] T 2af6372bb77e4533a38bc100eaba12e2 P c9688a355aa647069e7218293e5134c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0684b0d755ae4f238fbb08fd90659745 in term 1.
I20260929 16:41:55.145991 5793 leader_election.cc:302] T 2af6372bb77e4533a38bc100eaba12e2 P 0684b0d755ae4f238fbb08fd90659745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0684b0d755ae4f238fbb08fd90659745, c9688a355aa647069e7218293e5134c0; no voters:
I20260929 16:41:55.146339 5930 raft_consensus.cc:2781] T 2af6372bb77e4533a38bc100eaba12e2 P 0684b0d755ae4f238fbb08fd90659745 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:41:55.146840 5599 raft_consensus.cc:2454] T 2af6372bb77e4533a38bc100eaba12e2 P f3f4779c99f4471f9665f121adb398dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0684b0d755ae4f238fbb08fd90659745 in term 1.
I20260929 16:41:55.146912 5930 raft_consensus.cc:686] T 2af6372bb77e4533a38bc100eaba12e2 P 0684b0d755ae4f238fbb08fd90659745 [term 1 LEADER]: Becoming Leader. State: Replica: 0684b0d755ae4f238fbb08fd90659745, State: Running, Role: LEADER
I20260929 16:41:55.147585 5930 consensus_queue.cc:237] T 2af6372bb77e4533a38bc100eaba12e2 P 0684b0d755ae4f238fbb08fd90659745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9688a355aa647069e7218293e5134c0" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 40259 } } peers { permanent_uuid: "f3f4779c99f4471f9665f121adb398dc" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 40663 } } peers { permanent_uuid: "0684b0d755ae4f238fbb08fd90659745" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 44363 } }
I20260929 16:41:55.153214 5456 catalog_manager.cc:5495] T 2af6372bb77e4533a38bc100eaba12e2 P 0684b0d755ae4f238fbb08fd90659745 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0684b0d755ae4f238fbb08fd90659745 (127.0.197.67). New cstate: current_term: 1 leader_uuid: "0684b0d755ae4f238fbb08fd90659745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9688a355aa647069e7218293e5134c0" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 40259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3f4779c99f4471f9665f121adb398dc" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 40663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0684b0d755ae4f238fbb08fd90659745" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 44363 } health_report { overall_health: HEALTHY } } }
I20260929 16:41:55.188867 5921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:55.186421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50340 (local address 127.0.197.67:44363)
0929 16:41:55.186595 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:55.186599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:55.186620 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:41:55.187056 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:55.187060 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:55.187203 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:55.187305 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:55.188333 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:55.188337 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:55.188340 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:41:55.188355 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:55.188385 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:41:55.188400 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:55.188513 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:55.188597 (+ 84us) server_negotiation.cc:299] Negotiation successful
0929 16:41:55.188652 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 16:41:55.199205 5730 raft_consensus.cc:1261] T 2af6372bb77e4533a38bc100eaba12e2 P c9688a355aa647069e7218293e5134c0 [term 1 FOLLOWER]: Refusing update from remote peer 0684b0d755ae4f238fbb08fd90659745: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:41:55.199314 5599 raft_consensus.cc:1261] T 2af6372bb77e4533a38bc100eaba12e2 P f3f4779c99f4471f9665f121adb398dc [term 1 FOLLOWER]: Refusing update from remote peer 0684b0d755ae4f238fbb08fd90659745: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:41:55.200536 5933 consensus_queue.cc:1035] T 2af6372bb77e4533a38bc100eaba12e2 P 0684b0d755ae4f238fbb08fd90659745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9688a355aa647069e7218293e5134c0" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 40259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:41:55.200994 5930 consensus_queue.cc:1035] T 2af6372bb77e4533a38bc100eaba12e2 P 0684b0d755ae4f238fbb08fd90659745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3f4779c99f4471f9665f121adb398dc" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 40663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:41:55.214357 5936 mvcc.cc:205] Tried to move back new op lower bound from 7334707671841611776 to 7334707671649497088. Current Snapshot: MvccSnapshot[applied={T|T < 7334707671841611776}]
I20260929 16:41:55.224586 5937 mvcc.cc:205] Tried to move back new op lower bound from 7334707671841611776 to 7334707671649497088. Current Snapshot: MvccSnapshot[applied={T|T < 7334707671841611776}]
I20260929 16:41:55.238924 5458 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:41:55.239259 5458 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:41:55.243978 5458 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e99dcbdc84474c229dfaf49a99ade2dc: Sending DeleteTablet for 3 replicas of tablet 2af6372bb77e4533a38bc100eaba12e2
I20260929 16:41:55.245349 5840 tablet_service.cc:1514] Processing DeleteTablet for tablet 2af6372bb77e4533a38bc100eaba12e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:41:55 UTC) from {username='slave'} at 127.0.0.1:50326
I20260929 16:41:55.245344 5710 tablet_service.cc:1514] Processing DeleteTablet for tablet 2af6372bb77e4533a38bc100eaba12e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:41:55 UTC) from {username='slave'} at 127.0.0.1:50494
I20260929 16:41:55.246170 5579 tablet_service.cc:1514] Processing DeleteTablet for tablet 2af6372bb77e4533a38bc100eaba12e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:41:55 UTC) from {username='slave'} at 127.0.0.1:58236
I20260929 16:41:55.246327 5946 tablet_replica.cc:330] T 2af6372bb77e4533a38bc100eaba12e2 P c9688a355aa647069e7218293e5134c0: stopping tablet replica
I20260929 16:41:55.246809 5946 raft_consensus.cc:2229] T 2af6372bb77e4533a38bc100eaba12e2 P c9688a355aa647069e7218293e5134c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:41:55.247394 5946 raft_consensus.cc:2258] T 2af6372bb77e4533a38bc100eaba12e2 P c9688a355aa647069e7218293e5134c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:41:55.248395 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 5517
I20260929 16:41:55.249822 5946 ts_tablet_manager.cc:1905] T 2af6372bb77e4533a38bc100eaba12e2 P c9688a355aa647069e7218293e5134c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:41:55.255555 5948 tablet_replica.cc:330] T 2af6372bb77e4533a38bc100eaba12e2 P f3f4779c99f4471f9665f121adb398dc: stopping tablet replica
I20260929 16:41:55.256079 5948 raft_consensus.cc:2229] T 2af6372bb77e4533a38bc100eaba12e2 P f3f4779c99f4471f9665f121adb398dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:41:55.256646 5948 raft_consensus.cc:2258] T 2af6372bb77e4533a38bc100eaba12e2 P f3f4779c99f4471f9665f121adb398dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:41:55.258162 5946 ts_tablet_manager.cc:1918] T 2af6372bb77e4533a38bc100eaba12e2 P c9688a355aa647069e7218293e5134c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:41:55.258392 5946 log.cc:1196] T 2af6372bb77e4533a38bc100eaba12e2 P c9688a355aa647069e7218293e5134c0: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/ts-1/wal/wals/2af6372bb77e4533a38bc100eaba12e2
I20260929 16:41:55.258931 5946 ts_tablet_manager.cc:1939] T 2af6372bb77e4533a38bc100eaba12e2 P c9688a355aa647069e7218293e5134c0: Deleting consensus metadata
I20260929 16:41:55.259575 5947 tablet_replica.cc:330] T 2af6372bb77e4533a38bc100eaba12e2 P 0684b0d755ae4f238fbb08fd90659745: stopping tablet replica
I20260929 16:41:55.259837 5948 ts_tablet_manager.cc:1905] T 2af6372bb77e4533a38bc100eaba12e2 P f3f4779c99f4471f9665f121adb398dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:41:55.260054 5947 raft_consensus.cc:2229] T 2af6372bb77e4533a38bc100eaba12e2 P 0684b0d755ae4f238fbb08fd90659745 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:41:55.260777 5947 raft_consensus.cc:2258] T 2af6372bb77e4533a38bc100eaba12e2 P 0684b0d755ae4f238fbb08fd90659745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:41:55.262725 5947 ts_tablet_manager.cc:1905] T 2af6372bb77e4533a38bc100eaba12e2 P 0684b0d755ae4f238fbb08fd90659745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:41:55.268501 5919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:55.251193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58252 (local address 127.0.197.65:40663)
0929 16:41:55.251455 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:55.251464 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:55.262993 (+ 11529us) server_negotiation.cc:407] Connection header received
0929 16:41:55.263279 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:55.263285 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:55.263379 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:55.263481 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:55.267589 (+ 4108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:55.267596 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:55.267602 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:41:55.267627 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:55.267670 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:41:55.267689 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:55.267827 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:55.268089 (+ 262us) server_negotiation.cc:299] Negotiation successful
0929 16:41:55.268159 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260929 16:41:55.269160 5948 ts_tablet_manager.cc:1918] T 2af6372bb77e4533a38bc100eaba12e2 P f3f4779c99f4471f9665f121adb398dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:41:55.269379 5948 log.cc:1196] T 2af6372bb77e4533a38bc100eaba12e2 P f3f4779c99f4471f9665f121adb398dc: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/ts-0/wal/wals/2af6372bb77e4533a38bc100eaba12e2
I20260929 16:41:55.270092 5639 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:41:55.271818 5442 catalog_manager.cc:4841] TS c9688a355aa647069e7218293e5134c0 (127.0.197.66:40259): tablet 2af6372bb77e4533a38bc100eaba12e2 (table test-table [id=4ea8991b1c2f49868a031f8075ac7962]) successfully deleted
I20260929 16:41:55.272528 5947 ts_tablet_manager.cc:1918] T 2af6372bb77e4533a38bc100eaba12e2 P 0684b0d755ae4f238fbb08fd90659745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:41:55.272766 5947 log.cc:1196] T 2af6372bb77e4533a38bc100eaba12e2 P 0684b0d755ae4f238fbb08fd90659745: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790700068255611-789-0/minicluster-data/ts-2/wal/wals/2af6372bb77e4533a38bc100eaba12e2
I20260929 16:41:55.273404 5947 ts_tablet_manager.cc:1939] T 2af6372bb77e4533a38bc100eaba12e2 P 0684b0d755ae4f238fbb08fd90659745: Deleting consensus metadata
I20260929 16:41:55.275609 5443 catalog_manager.cc:4841] TS 0684b0d755ae4f238fbb08fd90659745 (127.0.197.67:44363): tablet 2af6372bb77e4533a38bc100eaba12e2 (table test-table [id=4ea8991b1c2f49868a031f8075ac7962]) successfully deleted
I20260929 16:41:55.363696 5948 ts_tablet_manager.cc:1939] T 2af6372bb77e4533a38bc100eaba12e2 P f3f4779c99f4471f9665f121adb398dc: Deleting consensus metadata
I20260929 16:41:55.364434 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 5517
W20260929 16:41:55.379586 5444 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.197.65:40663 (error 108)
I20260929 16:41:55.379799 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 5648
I20260929 16:41:55.381938 5922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:55.381210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:40663 (local address 127.0.0.1:58258)
0929 16:41:55.381546 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:55.381713 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.65:40663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191}
W20260929 16:41:55.382519 5444 catalog_manager.cc:4584] TS f3f4779c99f4471f9665f121adb398dc (127.0.197.65:40663): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2af6372bb77e4533a38bc100eaba12e2: Network error: Client connection negotiation failed: client connection to 127.0.197.65:40663: connect: Connection refused (error 111)
I20260929 16:41:55.383867 5920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:55.381411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50508 (local address 127.0.197.66:40259)
0929 16:41:55.381736 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:55.381741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:55.381758 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:41:55.381814 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:55.381818 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:55.381915 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:55.382055 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:55.383209 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:55.383215 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:55.383224 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:41:55.383241 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:55.383271 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:41:55.383283 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:55.383425 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:55.383512 (+ 87us) server_negotiation.cc:299] Negotiation successful
0929 16:41:55.383575 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260929 16:41:55.384848 5770 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:41:55.436578 5922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:55.435990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:40663 (local address 127.0.0.1:58260)
0929 16:41:55.436263 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:55.436379 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.65:40663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
I20260929 16:41:55.438025 5922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:55.437545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:40663 (local address 127.0.0.1:58272)
0929 16:41:55.437769 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:55.437867 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.65:40663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260929 16:41:55.498831 5922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:55.498169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:40663 (local address 127.0.0.1:58274)
0929 16:41:55.498497 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:55.498629 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.65:40663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196}
I20260929 16:41:55.500365 5922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:55.499828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:40663 (local address 127.0.0.1:58282)
0929 16:41:55.500075 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:55.500172 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.65:40663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
I20260929 16:41:55.510231 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 5648
I20260929 16:41:55.537585 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 5778
I20260929 16:41:55.541898 5921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:55.538400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50350 (local address 127.0.197.67:44363)
0929 16:41:55.538579 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:55.538584 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:55.538660 (+ 76us) server_negotiation.cc:407] Connection header received
0929 16:41:55.538925 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:55.538929 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:55.538994 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:55.539507 (+ 513us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:55.540715 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:55.540722 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:55.540726 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:41:55.540757 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:55.540811 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:41:55.540828 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:55.541044 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:55.541212 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 16:41:55.541288 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 16:41:55.542852 5900 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:41:55.581875 5922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:55.581175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:40663 (local address 127.0.0.1:58294)
0929 16:41:55.581525 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:55.581679 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.65:40663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260929 16:41:55.583429 5922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:55.582885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:40663 (local address 127.0.0.1:58308)
0929 16:41:55.583176 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:55.583271 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.65:40663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260929 16:41:55.666185 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 5778
I20260929 16:41:55.686342 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 5427
I20260929 16:41:55.687752 5488 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:41:55.797189 5922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:55.796428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:40663 (local address 127.0.0.1:58310)
0929 16:41:55.796834 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:55.796979 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.65:40663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225}
I20260929 16:41:55.797168 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 5427
2026-09-29T16:41:55Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2880 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260929 16:41:55.841799 789 mini_oidc.cc:152] Starting JWKS server
I20260929 16:41:55.844684 789 webserver.cc:460] Webserver started at https://127.0.0.1:40357/ using document root <none> and password file <none>
I20260929 16:41:55.845132 789 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 16:41:55.845875 789 webserver.cc:460] Webserver started at http://127.0.0.1:38895/ using document root <none> and password file <none>
2026-09-29T16:41:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:41:55Z Disabled control of system clock
I20260929 16:41:55.878439 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:44199
--webserver_interface=127.0.197.126
--webserver_port=0
--builtin_ntp_servers=127.0.197.84:40999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.126:44199
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40357/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790700068255611-789-0/testchainca.cert with env {}
W20260929 16:41:56.239486 5960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:41:56.239885 5960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:41:56.240057 5960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:41:56.247923 5960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:41:56.248090 5960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:41:56.248178 5960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:41:56.248283 5960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:41:56.248370 5960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 16:41:56.248464 5960 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 16:41:56.248541 5960 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40357/jwks.json
I20260929 16:41:56.259114 5960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:40999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790700068255611-789-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.197.126:44199
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:44199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40357/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.197.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790700068255611-789-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:41:56.260612 5960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:41:56.262013 5960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:41:56.275215 5965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:56.275187 5968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:56.275246 5966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:41:56.276202 5960 server_base.cc:1034] running on GCE node
I20260929 16:41:56.277125 5960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:41:56.278709 5960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:41:56.279973 5960 hybrid_clock.cc:648] HybridClock initialized: now 1790700116279900 us; error 65 us; skew 500 ppm
I20260929 16:41:56.280414 5960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:41:56.284416 5960 webserver.cc:460] Webserver started at http://127.0.197.126:41821/ using document root <none> and password file <none>
I20260929 16:41:56.285300 5960 fs_manager.cc:359] Metadata directory not provided
I20260929 16:41:56.285411 5960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:41:56.285787 5960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:41:56.289081 5960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790700068255611-789-0/minicluster-data/master-0/data/instance:
uuid: "04e6d22ec879422f8f7342185a105250"
format_stamp: "Formatted at 2026-09-29 16:41:56 on dist-test-slave-l31f"
server_key: "21e49f8a9cc48eb235120ca58684c592"
server_key_iv: "bfc0202680100f145e9727bdbb3b4da5"
server_key_version: "encryptionkey@0"
I20260929 16:41:56.290014 5960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790700068255611-789-0/minicluster-data/master-0/wal/instance:
uuid: "04e6d22ec879422f8f7342185a105250"
format_stamp: "Formatted at 2026-09-29 16:41:56 on dist-test-slave-l31f"
server_key: "21e49f8a9cc48eb235120ca58684c592"
server_key_iv: "bfc0202680100f145e9727bdbb3b4da5"
server_key_version: "encryptionkey@0"
I20260929 16:41:56.296481 5960 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260929 16:41:56.300607 5974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:56.302515 5960 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:41:56.302799 5960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "04e6d22ec879422f8f7342185a105250"
format_stamp: "Formatted at 2026-09-29 16:41:56 on dist-test-slave-l31f"
server_key: "21e49f8a9cc48eb235120ca58684c592"
server_key_iv: "bfc0202680100f145e9727bdbb3b4da5"
server_key_version: "encryptionkey@0"
I20260929 16:41:56.303268 5960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:41:56.323895 5960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:41:56.324903 5960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:41:56.325258 5960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:41:56.347150 5960 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.126:44199
I20260929 16:41:56.347414 6025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.126:44199 every 8 connection(s)
I20260929 16:41:56.348673 5960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:41:56.352876 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 5960
I20260929 16:41:56.353134 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790700068255611-789-0/minicluster-data/master-0/wal/instance
I20260929 16:41:56.353129 6026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:41:56.353929 789 external_mini_cluster.cc:1428] Setting key 0bceb5a0b6eea4981f38268facaeefb8
I20260929 16:41:56.369592 6026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04e6d22ec879422f8f7342185a105250: Bootstrap starting.
I20260929 16:41:56.373955 6029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:56.356221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53954 (local address 127.0.197.126:44199)
0929 16:41:56.357386 (+ 1165us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:56.357398 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:56.357445 (+ 47us) server_negotiation.cc:407] Connection header received
0929 16:41:56.360129 (+ 2684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:56.360179 (+ 50us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:56.361140 (+ 961us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:56.361564 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:56.362248 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:56.364398 (+ 2150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:56.366821 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:56.368337 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:56.368570 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:56.370150 (+ 1580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:56.370165 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:56.370197 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:41:56.371192 (+ 995us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:56.371323 (+ 131us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:41:56.371357 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:56.371493 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:56.372233 (+ 740us) server_negotiation.cc:299] Negotiation successful
0929 16:41:56.372636 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":426,"threads_started":1}
I20260929 16:41:56.374363 6026 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04e6d22ec879422f8f7342185a105250: Neither blocks nor log segments found. Creating new log.
I20260929 16:41:56.375594 6026 log.cc:824] T 00000000000000000000000000000000 P 04e6d22ec879422f8f7342185a105250: Log is configured to *not* fsync() on all Append() calls
I20260929 16:41:56.378371 789 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:41:56.379173 6026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04e6d22ec879422f8f7342185a105250: No bootstrap required, opened a new log
I20260929 16:41:56.387997 6026 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04e6d22ec879422f8f7342185a105250 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04e6d22ec879422f8f7342185a105250" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 44199 } }
I20260929 16:41:56.388437 6026 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04e6d22ec879422f8f7342185a105250 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:41:56.388540 6026 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04e6d22ec879422f8f7342185a105250 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04e6d22ec879422f8f7342185a105250, State: Initialized, Role: FOLLOWER
I20260929 16:41:56.389298 6026 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04e6d22ec879422f8f7342185a105250 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04e6d22ec879422f8f7342185a105250" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 44199 } }
I20260929 16:41:56.389601 6026 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04e6d22ec879422f8f7342185a105250 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:41:56.389751 6026 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04e6d22ec879422f8f7342185a105250 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:41:56.389966 6026 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04e6d22ec879422f8f7342185a105250 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:41:56.392282 6026 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04e6d22ec879422f8f7342185a105250 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04e6d22ec879422f8f7342185a105250" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 44199 } }
I20260929 16:41:56.392876 6026 leader_election.cc:302] T 00000000000000000000000000000000 P 04e6d22ec879422f8f7342185a105250 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04e6d22ec879422f8f7342185a105250; no voters:
I20260929 16:41:56.393563 6026 leader_election.cc:288] T 00000000000000000000000000000000 P 04e6d22ec879422f8f7342185a105250 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:41:56.393734 6031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04e6d22ec879422f8f7342185a105250 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:41:56.394505 6031 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04e6d22ec879422f8f7342185a105250 [term 1 LEADER]: Becoming Leader. State: Replica: 04e6d22ec879422f8f7342185a105250, State: Running, Role: LEADER
I20260929 16:41:56.395093 6031 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04e6d22ec879422f8f7342185a105250 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04e6d22ec879422f8f7342185a105250" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 44199 } }
I20260929 16:41:56.395933 6026 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04e6d22ec879422f8f7342185a105250 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:41:56.399323 6033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04e6d22ec879422f8f7342185a105250 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04e6d22ec879422f8f7342185a105250. Latest consensus state: current_term: 1 leader_uuid: "04e6d22ec879422f8f7342185a105250" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04e6d22ec879422f8f7342185a105250" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 44199 } } }
I20260929 16:41:56.399682 6033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04e6d22ec879422f8f7342185a105250 [sys.catalog]: This master's current role is: LEADER
I20260929 16:41:56.400578 6038 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:41:56.400396 6032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04e6d22ec879422f8f7342185a105250 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04e6d22ec879422f8f7342185a105250" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04e6d22ec879422f8f7342185a105250" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 44199 } } }
I20260929 16:41:56.400743 6032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04e6d22ec879422f8f7342185a105250 [sys.catalog]: This master's current role is: LEADER
I20260929 16:41:56.406805 6038 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:41:56.417040 6038 catalog_manager.cc:1347] Generated new cluster ID: b2c0ade0e21c41d081b179350131d190
I20260929 16:41:56.417174 6038 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:41:56.427987 6038 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:41:56.429364 6038 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:41:56.442247 6038 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04e6d22ec879422f8f7342185a105250: Generated new TSK 0
I20260929 16:41:56.443308 6038 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 16:41:56.485994 6056 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260929 16:41:56.486410 6056 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260929 16:41:56.488716 6055 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:41:56.461740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:44199 (local address 127.0.0.1:53962)
0929 16:41:56.462496 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:56.462518 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:56.462844 (+ 326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:56.463468 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:56.463472 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:56.463481 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0929 16:41:56.463831 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:56.463840 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:56.465426 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:56.465430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:56.466623 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:56.466635 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:56.468035 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:56.468041 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:56.468199 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:56.468282 (+ 83us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 16:41:56.486897 (+ 18615us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40357/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 16:41:56.487279 (+ 382us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.197.126:44199: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40357/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":369,"threads_started":1}
I20260929 16:41:56.489653 6029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:56.461912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53962 (local address 127.0.197.126:44199)
0929 16:41:56.462129 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:56.462136 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:56.462648 (+ 512us) server_negotiation.cc:407] Connection header received
0929 16:41:56.463134 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:56.463143 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:56.463256 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:56.463442 (+ 186us) server_negotiation.cc:226] Negotiated authn=JWT
0929 16:41:56.464004 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:56.465230 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:56.466802 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:56.467618 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:56.467777 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:56.468664 (+ 887us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 16:41:56.486501 (+ 17837us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40357/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 16:41:56.489317 (+ 2816us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53962: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40357/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":87}
W20260929 16:41:56.489790 6029 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53962: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40357/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.197.126:44199: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40357/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260929 16:41:56.490283 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 5960
I20260929 16:41:56.491825 6021 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:41:56.587515 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 5960
2026-09-29T16:41:56Z chronyd exiting
I20260929 16:41:56.638659 789 test_util.cc:175] -----------------------------------------------
I20260929 16:41:56.638830 789 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790700068255611-789-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (801 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260929 16:41:56.642858 789 mini_oidc.cc:152] Starting JWKS server
I20260929 16:41:56.645087 789 webserver.cc:460] Webserver started at https://127.0.0.1:39761/ using document root <none> and password file <none>
I20260929 16:41:56.645382 789 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 16:41:56.646128 789 webserver.cc:460] Webserver started at http://127.0.0.1:46079/ using document root <none> and password file <none>
2026-09-29T16:41:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:41:56Z Disabled control of system clock
I20260929 16:41:56.676028 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:39877
--webserver_interface=127.0.197.126
--webserver_port=0
--builtin_ntp_servers=127.0.197.84:39503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.126:39877
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39761/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790700068255611-789-0/testchainca.cert with env {}
W20260929 16:41:57.022701 6063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:41:57.023161 6063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:41:57.023268 6063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:41:57.031235 6063 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:41:57.031378 6063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:41:57.031416 6063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:41:57.031468 6063 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:41:57.031512 6063 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 16:41:57.031576 6063 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 16:41:57.031611 6063 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39761/jwks.json
I20260929 16:41:57.042245 6063 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:39503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790700068255611-789-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.197.126:39877
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:39877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39761/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.197.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790700068255611-789-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:41:57.043766 6063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:41:57.045081 6063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:41:57.057657 6071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:57.057626 6068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:57.057821 6069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:41:57.058502 6063 server_base.cc:1034] running on GCE node
I20260929 16:41:57.059507 6063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:41:57.061101 6063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:41:57.062319 6063 hybrid_clock.cc:648] HybridClock initialized: now 1790700117062284 us; error 39 us; skew 500 ppm
I20260929 16:41:57.062726 6063 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:41:57.065488 6063 webserver.cc:460] Webserver started at http://127.0.197.126:39265/ using document root <none> and password file <none>
I20260929 16:41:57.066283 6063 fs_manager.cc:359] Metadata directory not provided
I20260929 16:41:57.066425 6063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:41:57.066845 6063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:41:57.069975 6063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790700068255611-789-0/minicluster-data/master-0/data/instance:
uuid: "4dff808d12564c568dfc7d80f0b8ee49"
format_stamp: "Formatted at 2026-09-29 16:41:57 on dist-test-slave-l31f"
server_key: "02bbc16fe89cd4822dc47fd650d62649"
server_key_iv: "00a4ac3dc7f6e74cfa50b7837b688b8f"
server_key_version: "encryptionkey@0"
I20260929 16:41:57.070892 6063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790700068255611-789-0/minicluster-data/master-0/wal/instance:
uuid: "4dff808d12564c568dfc7d80f0b8ee49"
format_stamp: "Formatted at 2026-09-29 16:41:57 on dist-test-slave-l31f"
server_key: "02bbc16fe89cd4822dc47fd650d62649"
server_key_iv: "00a4ac3dc7f6e74cfa50b7837b688b8f"
server_key_version: "encryptionkey@0"
I20260929 16:41:57.076598 6063 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 16:41:57.080615 6077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:57.082149 6063 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:41:57.082365 6063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "4dff808d12564c568dfc7d80f0b8ee49"
format_stamp: "Formatted at 2026-09-29 16:41:57 on dist-test-slave-l31f"
server_key: "02bbc16fe89cd4822dc47fd650d62649"
server_key_iv: "00a4ac3dc7f6e74cfa50b7837b688b8f"
server_key_version: "encryptionkey@0"
I20260929 16:41:57.082635 6063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:41:57.096131 6063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:41:57.097054 6063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:41:57.097409 6063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:41:57.116722 6063 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.126:39877
I20260929 16:41:57.116748 6128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.126:39877 every 8 connection(s)
I20260929 16:41:57.118327 6063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:41:57.122291 6129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:41:57.128283 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 6063
I20260929 16:41:57.128594 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790700068255611-789-0/minicluster-data/master-0/wal/instance
I20260929 16:41:57.129238 789 external_mini_cluster.cc:1428] Setting key 2891eb45c2b6fea807ee55fc7afc0c63
I20260929 16:41:57.134441 6129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dff808d12564c568dfc7d80f0b8ee49: Bootstrap starting.
I20260929 16:41:57.137929 6129 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4dff808d12564c568dfc7d80f0b8ee49: Neither blocks nor log segments found. Creating new log.
I20260929 16:41:57.139156 6129 log.cc:824] T 00000000000000000000000000000000 P 4dff808d12564c568dfc7d80f0b8ee49: Log is configured to *not* fsync() on all Append() calls
I20260929 16:41:57.142333 6129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dff808d12564c568dfc7d80f0b8ee49: No bootstrap required, opened a new log
I20260929 16:41:57.143389 6132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:57.131320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38942 (local address 127.0.197.126:39877)
0929 16:41:57.132472 (+ 1152us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:57.132484 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:57.132530 (+ 46us) server_negotiation.cc:407] Connection header received
0929 16:41:57.134295 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:57.134319 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:57.134892 (+ 573us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:57.135366 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:57.135947 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:57.137239 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:57.138606 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:57.139487 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:57.139703 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:57.140903 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:57.140910 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:57.140927 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:41:57.141490 (+ 563us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:57.141569 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:41:57.141592 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:57.141726 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:57.141965 (+ 239us) server_negotiation.cc:299] Negotiation successful
0929 16:41:57.142293 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":371,"threads_started":1}
I20260929 16:41:57.147927 789 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:41:57.151445 6129 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4dff808d12564c568dfc7d80f0b8ee49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dff808d12564c568dfc7d80f0b8ee49" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 39877 } }
I20260929 16:41:57.151859 6129 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4dff808d12564c568dfc7d80f0b8ee49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:41:57.151984 6129 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4dff808d12564c568dfc7d80f0b8ee49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dff808d12564c568dfc7d80f0b8ee49, State: Initialized, Role: FOLLOWER
I20260929 16:41:57.152581 6129 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4dff808d12564c568dfc7d80f0b8ee49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dff808d12564c568dfc7d80f0b8ee49" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 39877 } }
I20260929 16:41:57.152866 6129 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4dff808d12564c568dfc7d80f0b8ee49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:41:57.153026 6129 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4dff808d12564c568dfc7d80f0b8ee49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:41:57.153225 6129 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4dff808d12564c568dfc7d80f0b8ee49 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:41:57.155723 6129 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4dff808d12564c568dfc7d80f0b8ee49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dff808d12564c568dfc7d80f0b8ee49" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 39877 } }
I20260929 16:41:57.156388 6129 leader_election.cc:302] T 00000000000000000000000000000000 P 4dff808d12564c568dfc7d80f0b8ee49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4dff808d12564c568dfc7d80f0b8ee49; no voters:
I20260929 16:41:57.156994 6129 leader_election.cc:288] T 00000000000000000000000000000000 P 4dff808d12564c568dfc7d80f0b8ee49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:41:57.157189 6134 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4dff808d12564c568dfc7d80f0b8ee49 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:41:57.157896 6134 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4dff808d12564c568dfc7d80f0b8ee49 [term 1 LEADER]: Becoming Leader. State: Replica: 4dff808d12564c568dfc7d80f0b8ee49, State: Running, Role: LEADER
I20260929 16:41:57.158465 6134 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4dff808d12564c568dfc7d80f0b8ee49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dff808d12564c568dfc7d80f0b8ee49" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 39877 } }
I20260929 16:41:57.159379 6129 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4dff808d12564c568dfc7d80f0b8ee49 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:41:57.162010 6135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dff808d12564c568dfc7d80f0b8ee49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4dff808d12564c568dfc7d80f0b8ee49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dff808d12564c568dfc7d80f0b8ee49" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 39877 } } }
I20260929 16:41:57.162117 6136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dff808d12564c568dfc7d80f0b8ee49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4dff808d12564c568dfc7d80f0b8ee49. Latest consensus state: current_term: 1 leader_uuid: "4dff808d12564c568dfc7d80f0b8ee49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dff808d12564c568dfc7d80f0b8ee49" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 39877 } } }
I20260929 16:41:57.162448 6135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dff808d12564c568dfc7d80f0b8ee49 [sys.catalog]: This master's current role is: LEADER
I20260929 16:41:57.162515 6136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dff808d12564c568dfc7d80f0b8ee49 [sys.catalog]: This master's current role is: LEADER
I20260929 16:41:57.172402 6139 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:41:57.178644 6139 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:41:57.188944 6139 catalog_manager.cc:1347] Generated new cluster ID: f8d357a162aa44139da62c504896fb4e
I20260929 16:41:57.189117 6139 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:41:57.203474 6139 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:41:57.205173 6139 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:41:57.226511 6139 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4dff808d12564c568dfc7d80f0b8ee49: Generated new TSK 0
I20260929 16:41:57.227660 6139 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 16:41:57.269315 6159 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260929 16:41:57.269476 6159 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 16:41:57.270778 6132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:57.249760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38948 (local address 127.0.197.126:39877)
0929 16:41:57.249971 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:57.249979 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:57.250344 (+ 365us) server_negotiation.cc:407] Connection header received
0929 16:41:57.250577 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:57.250584 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:57.250682 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:57.250885 (+ 203us) server_negotiation.cc:226] Negotiated authn=JWT
0929 16:41:57.251707 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:57.252885 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:57.254303 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:57.255075 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:57.255220 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:57.255719 (+ 499us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 16:41:57.270019 (+ 14300us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39761/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 16:41:57.270470 (+ 451us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38948: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39761/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":78}
W20260929 16:41:57.270916 6132 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38948: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39761/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260929 16:41:57.270913 6158 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:41:57.249569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:39877 (local address 127.0.0.1:38948)
0929 16:41:57.250231 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:57.250248 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:57.250383 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:57.250945 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:57.250949 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:57.250958 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0929 16:41:57.251493 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:57.251506 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:57.253082 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:57.253086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:57.254142 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:57.254154 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:57.255244 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:57.255249 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:57.255396 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:57.255432 (+ 36us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 16:41:57.270323 (+ 14891us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39761/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 16:41:57.270563 (+ 240us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.197.126:39877: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39761/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":374,"threads_started":1}
I20260929 16:41:57.272517 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 6063
I20260929 16:41:57.273993 6124 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:41:57.370954 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 6063
2026-09-29T16:41:57Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (785 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260929 16:41:57.427874 789 mini_oidc.cc:152] Starting JWKS server
I20260929 16:41:57.429801 789 webserver.cc:460] Webserver started at https://127.0.0.1:35639/ using document root <none> and password file <none>
I20260929 16:41:57.430022 789 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 16:41:57.430660 789 webserver.cc:460] Webserver started at http://127.0.0.1:36629/ using document root <none> and password file <none>
2026-09-29T16:41:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:41:57Z Disabled control of system clock
I20260929 16:41:57.460335 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:35435
--webserver_interface=127.0.197.126
--webserver_port=0
--builtin_ntp_servers=127.0.197.84:36169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.126:35435
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35639/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 16:41:57.804210 6166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:41:57.804543 6166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:41:57.804688 6166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:41:57.812197 6166 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:41:57.812327 6166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:41:57.812379 6166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:41:57.812511 6166 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:41:57.812578 6166 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 16:41:57.812696 6166 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 16:41:57.812759 6166 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35639/jwks.json
I20260929 16:41:57.823553 6166 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:36169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790700068255611-789-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.197.126:35435
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:35435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35639/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.197.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:41:57.825028 6166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:41:57.826993 6166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:41:57.840260 6172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:57.840175 6174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:57.840304 6171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:41:57.841331 6166 server_base.cc:1034] running on GCE node
I20260929 16:41:57.842273 6166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:41:57.844012 6166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:41:57.845256 6166 hybrid_clock.cc:648] HybridClock initialized: now 1790700117845211 us; error 44 us; skew 500 ppm
I20260929 16:41:57.845677 6166 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:41:57.848282 6166 webserver.cc:460] Webserver started at http://127.0.197.126:45665/ using document root <none> and password file <none>
I20260929 16:41:57.849110 6166 fs_manager.cc:359] Metadata directory not provided
I20260929 16:41:57.849216 6166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:41:57.849589 6166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:41:57.852917 6166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790700068255611-789-0/minicluster-data/master-0/data/instance:
uuid: "736bb7382fe745e69105f08c2ece5a76"
format_stamp: "Formatted at 2026-09-29 16:41:57 on dist-test-slave-l31f"
server_key: "608c533c6c3d1d791fdd0c9a9c9c78ee"
server_key_iv: "aaff01780070d7a1a65c4a6762be48a6"
server_key_version: "encryptionkey@0"
I20260929 16:41:57.853834 6166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790700068255611-789-0/minicluster-data/master-0/wal/instance:
uuid: "736bb7382fe745e69105f08c2ece5a76"
format_stamp: "Formatted at 2026-09-29 16:41:57 on dist-test-slave-l31f"
server_key: "608c533c6c3d1d791fdd0c9a9c9c78ee"
server_key_iv: "aaff01780070d7a1a65c4a6762be48a6"
server_key_version: "encryptionkey@0"
I20260929 16:41:57.859687 6166 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260929 16:41:57.863536 6180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:57.864997 6166 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:41:57.865201 6166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "736bb7382fe745e69105f08c2ece5a76"
format_stamp: "Formatted at 2026-09-29 16:41:57 on dist-test-slave-l31f"
server_key: "608c533c6c3d1d791fdd0c9a9c9c78ee"
server_key_iv: "aaff01780070d7a1a65c4a6762be48a6"
server_key_version: "encryptionkey@0"
I20260929 16:41:57.865440 6166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:41:57.889078 6166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:41:57.890080 6166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:41:57.890439 6166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:41:57.910837 6166 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.126:35435
I20260929 16:41:57.910877 6231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.126:35435 every 8 connection(s)
I20260929 16:41:57.912377 6166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:41:57.916410 6232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:41:57.922681 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 6166
I20260929 16:41:57.922930 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790700068255611-789-0/minicluster-data/master-0/wal/instance
I20260929 16:41:57.923787 789 external_mini_cluster.cc:1428] Setting key 4aa679164617375335f726b0b6b652c4
I20260929 16:41:57.928942 6232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 736bb7382fe745e69105f08c2ece5a76: Bootstrap starting.
I20260929 16:41:57.932461 6232 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 736bb7382fe745e69105f08c2ece5a76: Neither blocks nor log segments found. Creating new log.
I20260929 16:41:57.933558 6232 log.cc:824] T 00000000000000000000000000000000 P 736bb7382fe745e69105f08c2ece5a76: Log is configured to *not* fsync() on all Append() calls
I20260929 16:41:57.936972 6232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 736bb7382fe745e69105f08c2ece5a76: No bootstrap required, opened a new log
I20260929 16:41:57.938627 6235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:57.925977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35644 (local address 127.0.197.126:35435)
0929 16:41:57.927192 (+ 1215us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:57.927205 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:57.927254 (+ 49us) server_negotiation.cc:407] Connection header received
0929 16:41:57.929014 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:57.929076 (+ 62us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:57.929633 (+ 557us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:57.930050 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:57.930684 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:57.932102 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:57.933585 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:57.934584 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:57.934885 (+ 301us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:57.936161 (+ 1276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:57.936171 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:57.936190 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:41:57.936767 (+ 577us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:57.936860 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:41:57.936885 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:57.937026 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:57.937256 (+ 230us) server_negotiation.cc:299] Negotiation successful
0929 16:41:57.937628 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":377,"threads_started":1}
I20260929 16:41:57.943172 789 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:41:57.945421 6232 raft_consensus.cc:348] T 00000000000000000000000000000000 P 736bb7382fe745e69105f08c2ece5a76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "736bb7382fe745e69105f08c2ece5a76" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 35435 } }
I20260929 16:41:57.945933 6232 raft_consensus.cc:374] T 00000000000000000000000000000000 P 736bb7382fe745e69105f08c2ece5a76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:41:57.946065 6232 raft_consensus.cc:729] T 00000000000000000000000000000000 P 736bb7382fe745e69105f08c2ece5a76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 736bb7382fe745e69105f08c2ece5a76, State: Initialized, Role: FOLLOWER
I20260929 16:41:57.946723 6232 consensus_queue.cc:260] T 00000000000000000000000000000000 P 736bb7382fe745e69105f08c2ece5a76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "736bb7382fe745e69105f08c2ece5a76" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 35435 } }
I20260929 16:41:57.947067 6232 raft_consensus.cc:388] T 00000000000000000000000000000000 P 736bb7382fe745e69105f08c2ece5a76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:41:57.947206 6232 raft_consensus.cc:482] T 00000000000000000000000000000000 P 736bb7382fe745e69105f08c2ece5a76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:41:57.947338 6232 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 736bb7382fe745e69105f08c2ece5a76 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:41:57.949740 6232 raft_consensus.cc:504] T 00000000000000000000000000000000 P 736bb7382fe745e69105f08c2ece5a76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "736bb7382fe745e69105f08c2ece5a76" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 35435 } }
I20260929 16:41:57.950387 6232 leader_election.cc:302] T 00000000000000000000000000000000 P 736bb7382fe745e69105f08c2ece5a76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 736bb7382fe745e69105f08c2ece5a76; no voters:
I20260929 16:41:57.951195 6232 leader_election.cc:288] T 00000000000000000000000000000000 P 736bb7382fe745e69105f08c2ece5a76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:41:57.951359 6237 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 736bb7382fe745e69105f08c2ece5a76 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:41:57.952088 6237 raft_consensus.cc:686] T 00000000000000000000000000000000 P 736bb7382fe745e69105f08c2ece5a76 [term 1 LEADER]: Becoming Leader. State: Replica: 736bb7382fe745e69105f08c2ece5a76, State: Running, Role: LEADER
I20260929 16:41:57.952680 6237 consensus_queue.cc:237] T 00000000000000000000000000000000 P 736bb7382fe745e69105f08c2ece5a76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "736bb7382fe745e69105f08c2ece5a76" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 35435 } }
I20260929 16:41:57.953629 6232 sys_catalog.cc:564] T 00000000000000000000000000000000 P 736bb7382fe745e69105f08c2ece5a76 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:41:57.955706 6239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 736bb7382fe745e69105f08c2ece5a76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 736bb7382fe745e69105f08c2ece5a76. Latest consensus state: current_term: 1 leader_uuid: "736bb7382fe745e69105f08c2ece5a76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "736bb7382fe745e69105f08c2ece5a76" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 35435 } } }
I20260929 16:41:57.955732 6238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 736bb7382fe745e69105f08c2ece5a76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "736bb7382fe745e69105f08c2ece5a76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "736bb7382fe745e69105f08c2ece5a76" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 35435 } } }
I20260929 16:41:57.956163 6238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 736bb7382fe745e69105f08c2ece5a76 [sys.catalog]: This master's current role is: LEADER
I20260929 16:41:57.956163 6239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 736bb7382fe745e69105f08c2ece5a76 [sys.catalog]: This master's current role is: LEADER
I20260929 16:41:57.957329 6243 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:41:57.964989 6243 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:41:57.979532 6243 catalog_manager.cc:1347] Generated new cluster ID: 61d560ef3bb74048830b3b48c448438a
I20260929 16:41:57.979707 6243 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:41:57.990902 6243 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:41:57.992621 6243 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:41:58.002976 6243 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 736bb7382fe745e69105f08c2ece5a76: Generated new TSK 0
I20260929 16:41:58.004056 6243 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:41:58.037061 6235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:58.027341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35656 (local address 127.0.197.126:35435)
0929 16:41:58.027559 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:58.027566 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:58.028034 (+ 468us) server_negotiation.cc:407] Connection header received
0929 16:41:58.028265 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:58.028271 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:58.028384 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:58.028586 (+ 202us) server_negotiation.cc:226] Negotiated authn=JWT
0929 16:41:58.029083 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:58.030258 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:58.032155 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:58.032841 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:58.032982 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:58.033772 (+ 790us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 16:41:58.036337 (+ 2565us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35639/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 16:41:58.036708 (+ 371us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35656: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35639/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":72}
W20260929 16:41:58.037213 6235 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35656: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35639/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260929 16:41:58.039147 6261 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:41:58.027200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:35435 (local address 127.0.0.1:35656)
0929 16:41:58.027916 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:58.027933 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:58.028070 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:58.028601 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:58.028604 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:58.028614 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0929 16:41:58.028925 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:58.028933 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:58.030446 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:58.030450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:58.031987 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:58.032002 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:58.033140 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:58.033148 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:58.033389 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:58.033447 (+ 58us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 16:41:58.036584 (+ 3137us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35639/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 16:41:58.038384 (+ 1800us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.197.126:35435: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35639/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":540,"mutex_wait_us":1446,"thread_start_us":398,"threads_started":1}
I20260929 16:41:58.040534 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 6166
I20260929 16:41:58.041817 6227 generic_service.cc:196] Checking for leaks (request via RPC)
E20260929 16:41:58.053223 6262 webserver.cc:533] Webserver: sslize failed:
E20260929 16:41:58.053398 6262 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 16:41:58.137671 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 6166
2026-09-29T16:41:58Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (766 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-29T16:41:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:41:58Z Disabled control of system clock
I20260929 16:41:58.223394 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:39467
--webserver_interface=127.0.197.126
--webserver_port=0
--builtin_ntp_servers=127.0.197.84:35773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.126:39467
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790700068255611-789-0/testchainca.cert with env {}
W20260929 16:41:58.580137 6267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:41:58.580493 6267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:41:58.580658 6267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:41:58.588213 6267 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:41:58.588364 6267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:41:58.588403 6267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:41:58.588529 6267 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:41:58.588582 6267 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 16:41:58.588699 6267 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 16:41:58.588765 6267 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260929 16:41:58.599572 6267 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:35773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790700068255611-789-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.197.126:39467
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:39467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.197.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790700068255611-789-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:41:58.601104 6267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:41:58.602573 6267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:41:58.615335 6272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:58.615309 6273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:41:58.615294 6275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:41:58.616333 6267 server_base.cc:1034] running on GCE node
I20260929 16:41:58.617129 6267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:41:58.618690 6267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:41:58.619912 6267 hybrid_clock.cc:648] HybridClock initialized: now 1790700118619838 us; error 75 us; skew 500 ppm
I20260929 16:41:58.620258 6267 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:41:58.627429 6267 webserver.cc:460] Webserver started at http://127.0.197.126:41483/ using document root <none> and password file <none>
I20260929 16:41:58.628189 6267 fs_manager.cc:359] Metadata directory not provided
I20260929 16:41:58.628305 6267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:41:58.628575 6267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:41:58.631752 6267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790700068255611-789-0/minicluster-data/master-0/data/instance:
uuid: "694b93ad55384dd5bfdd7f92a18e4a3a"
format_stamp: "Formatted at 2026-09-29 16:41:58 on dist-test-slave-l31f"
server_key: "81401df0a550134464b2eee3ff06970f"
server_key_iv: "73eb959b2f08a4ca493155477fbaaa5f"
server_key_version: "encryptionkey@0"
I20260929 16:41:58.632611 6267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790700068255611-789-0/minicluster-data/master-0/wal/instance:
uuid: "694b93ad55384dd5bfdd7f92a18e4a3a"
format_stamp: "Formatted at 2026-09-29 16:41:58 on dist-test-slave-l31f"
server_key: "81401df0a550134464b2eee3ff06970f"
server_key_iv: "73eb959b2f08a4ca493155477fbaaa5f"
server_key_version: "encryptionkey@0"
I20260929 16:41:58.638314 6267 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 16:41:58.642517 6281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:41:58.644681 6267 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260929 16:41:58.644894 6267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "694b93ad55384dd5bfdd7f92a18e4a3a"
format_stamp: "Formatted at 2026-09-29 16:41:58 on dist-test-slave-l31f"
server_key: "81401df0a550134464b2eee3ff06970f"
server_key_iv: "73eb959b2f08a4ca493155477fbaaa5f"
server_key_version: "encryptionkey@0"
I20260929 16:41:58.645129 6267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:41:58.673254 6267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:41:58.674124 6267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:41:58.674382 6267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:41:58.695111 6267 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.126:39467
I20260929 16:41:58.695158 6332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.126:39467 every 8 connection(s)
I20260929 16:41:58.696575 6267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:41:58.700587 6333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:41:58.706913 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 6267
I20260929 16:41:58.707186 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790700068255611-789-0/minicluster-data/master-0/wal/instance
I20260929 16:41:58.707806 789 external_mini_cluster.cc:1428] Setting key ab6a37da8f7a396e4e98c4c9d52cbd25
I20260929 16:41:58.712677 6333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 694b93ad55384dd5bfdd7f92a18e4a3a: Bootstrap starting.
I20260929 16:41:58.716172 6333 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 694b93ad55384dd5bfdd7f92a18e4a3a: Neither blocks nor log segments found. Creating new log.
I20260929 16:41:58.717273 6333 log.cc:824] T 00000000000000000000000000000000 P 694b93ad55384dd5bfdd7f92a18e4a3a: Log is configured to *not* fsync() on all Append() calls
I20260929 16:41:58.721194 6333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 694b93ad55384dd5bfdd7f92a18e4a3a: No bootstrap required, opened a new log
I20260929 16:41:58.723151 6336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:58.709834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34666 (local address 127.0.197.126:39467)
0929 16:41:58.710975 (+ 1141us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:58.710988 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:58.711105 (+ 117us) server_negotiation.cc:407] Connection header received
0929 16:41:58.712817 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:58.712873 (+ 56us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:58.713423 (+ 550us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:58.713845 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:41:58.714465 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:58.715969 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:58.717367 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:58.718502 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:58.718712 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:58.720089 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:41:58.720103 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:41:58.720126 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:41:58.720902 (+ 776us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:41:58.721013 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:41:58.721041 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:41:58.721171 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 16:41:58.721413 (+ 242us) server_negotiation.cc:299] Negotiation successful
0929 16:41:58.721798 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":417,"threads_started":1}
I20260929 16:41:58.727664 789 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:41:58.730360 6333 raft_consensus.cc:348] T 00000000000000000000000000000000 P 694b93ad55384dd5bfdd7f92a18e4a3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "694b93ad55384dd5bfdd7f92a18e4a3a" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 39467 } }
I20260929 16:41:58.730753 6333 raft_consensus.cc:374] T 00000000000000000000000000000000 P 694b93ad55384dd5bfdd7f92a18e4a3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:41:58.730938 6333 raft_consensus.cc:729] T 00000000000000000000000000000000 P 694b93ad55384dd5bfdd7f92a18e4a3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 694b93ad55384dd5bfdd7f92a18e4a3a, State: Initialized, Role: FOLLOWER
I20260929 16:41:58.731626 6333 consensus_queue.cc:260] T 00000000000000000000000000000000 P 694b93ad55384dd5bfdd7f92a18e4a3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "694b93ad55384dd5bfdd7f92a18e4a3a" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 39467 } }
I20260929 16:41:58.731899 6333 raft_consensus.cc:388] T 00000000000000000000000000000000 P 694b93ad55384dd5bfdd7f92a18e4a3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:41:58.732043 6333 raft_consensus.cc:482] T 00000000000000000000000000000000 P 694b93ad55384dd5bfdd7f92a18e4a3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:41:58.732231 6333 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 694b93ad55384dd5bfdd7f92a18e4a3a [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:41:58.734504 6333 raft_consensus.cc:504] T 00000000000000000000000000000000 P 694b93ad55384dd5bfdd7f92a18e4a3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "694b93ad55384dd5bfdd7f92a18e4a3a" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 39467 } }
I20260929 16:41:58.735188 6333 leader_election.cc:302] T 00000000000000000000000000000000 P 694b93ad55384dd5bfdd7f92a18e4a3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 694b93ad55384dd5bfdd7f92a18e4a3a; no voters:
I20260929 16:41:58.735831 6333 leader_election.cc:288] T 00000000000000000000000000000000 P 694b93ad55384dd5bfdd7f92a18e4a3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:41:58.735994 6338 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 694b93ad55384dd5bfdd7f92a18e4a3a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:41:58.736703 6338 raft_consensus.cc:686] T 00000000000000000000000000000000 P 694b93ad55384dd5bfdd7f92a18e4a3a [term 1 LEADER]: Becoming Leader. State: Replica: 694b93ad55384dd5bfdd7f92a18e4a3a, State: Running, Role: LEADER
I20260929 16:41:58.737393 6338 consensus_queue.cc:237] T 00000000000000000000000000000000 P 694b93ad55384dd5bfdd7f92a18e4a3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "694b93ad55384dd5bfdd7f92a18e4a3a" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 39467 } }
I20260929 16:41:58.738255 6333 sys_catalog.cc:564] T 00000000000000000000000000000000 P 694b93ad55384dd5bfdd7f92a18e4a3a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:41:58.740262 6340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 694b93ad55384dd5bfdd7f92a18e4a3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 694b93ad55384dd5bfdd7f92a18e4a3a. Latest consensus state: current_term: 1 leader_uuid: "694b93ad55384dd5bfdd7f92a18e4a3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "694b93ad55384dd5bfdd7f92a18e4a3a" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 39467 } } }
I20260929 16:41:58.740662 6340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 694b93ad55384dd5bfdd7f92a18e4a3a [sys.catalog]: This master's current role is: LEADER
I20260929 16:41:58.740795 6339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 694b93ad55384dd5bfdd7f92a18e4a3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "694b93ad55384dd5bfdd7f92a18e4a3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "694b93ad55384dd5bfdd7f92a18e4a3a" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 39467 } } }
I20260929 16:41:58.741289 6339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 694b93ad55384dd5bfdd7f92a18e4a3a [sys.catalog]: This master's current role is: LEADER
I20260929 16:41:58.742062 6346 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:41:58.751863 6346 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:41:58.768280 6346 catalog_manager.cc:1347] Generated new cluster ID: e9c046630f7b457290d9f5f83502fea3
I20260929 16:41:58.768448 6346 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:41:58.784245 6346 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:41:58.785714 6346 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:41:58.796139 6346 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 694b93ad55384dd5bfdd7f92a18e4a3a: Generated new TSK 0
I20260929 16:41:58.797114 6346 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:41:58.916783 6336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:41:58.821086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34674 (local address 127.0.197.126:39467)
0929 16:41:58.821412 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0929 16:41:58.821420 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:41:58.821603 (+ 183us) server_negotiation.cc:407] Connection header received
0929 16:41:58.821789 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:41:58.821794 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:41:58.821897 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:41:58.822103 (+ 206us) server_negotiation.cc:226] Negotiated authn=JWT
0929 16:41:58.822624 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:58.823853 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:58.825516 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:41:58.826231 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:41:58.826380 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:58.827039 (+ 659us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 16:41:58.915926 (+ 88887us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 16:41:58.916363 (+ 437us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34674: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":144}
W20260929 16:41:58.916931 6362 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:41:58.820709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:39467 (local address 127.0.0.1:34674)
0929 16:41:58.821400 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0929 16:41:58.821420 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:41:58.821567 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:41:58.822146 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:41:58.822150 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:41:58.822162 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0929 16:41:58.822446 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:58.822455 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:58.824080 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:58.824085 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:58.825323 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:41:58.825339 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:41:58.826468 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:41:58.826474 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:41:58.826636 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:41:58.826676 (+ 40us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 16:41:58.916211 (+ 89535us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 16:41:58.916539 (+ 328us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.197.126:39467: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":348,"threads_started":1}
W20260929 16:41:58.916973 6336 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34674: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260929 16:41:58.918537 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 6267
I20260929 16:41:58.919852 6328 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:41:59.029775 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 6267
2026-09-29T16:41:59Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (871 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (439 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (432 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:42:00 dist-test-slave-l31f krb5kdc[6377](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:42:00 dist-test-slave-l31f krb5kdc[6377](info): set up 2 sockets
Sep 29 16:42:00 dist-test-slave-l31f krb5kdc[6377](info): commencing operation
krb5kdc: starting...
W20260929 16:42:01.023882 789 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/krb5kdc/test-user.keytab.
Sep 29 16:42:01 dist-test-slave-l31f krb5kdc[6377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:42:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:42:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.197.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
I20260929 16:42:01.256181 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:42305
--webserver_interface=127.0.197.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:35065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.126:42305
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:42:01.604735 6390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:42:01.605152 6390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:42:01.605330 6390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:42:01.612931 6390 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:42:01.613125 6390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:42:01.613241 6390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:42:01.613351 6390 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:42:01.613438 6390 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:42:01.623781 6390 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:35065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.197.126:42305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:42305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.197.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:42:01.625299 6390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:42:01.626806 6390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:42:01.640237 6396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:01.640270 6398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:01.640313 6395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:42:01.646634 6390 server_base.cc:1034] running on GCE node
I20260929 16:42:01.647647 6390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:42:01.649260 6390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:42:01.650498 6390 hybrid_clock.cc:648] HybridClock initialized: now 1790700121650429 us; error 63 us; skew 500 ppm
I20260929 16:42:01.650924 6390 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:42:01 dist-test-slave-l31f krb5kdc[6377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:42:01.654582 6390 init.cc:377] Logged in from keytab as kudu/127.0.197.126@KRBTEST.COM (short username kudu)
I20260929 16:42:01.656311 6390 webserver.cc:460] Webserver started at http://127.0.197.126:40845/ using document root <none> and password file <none>
I20260929 16:42:01.657050 6390 fs_manager.cc:359] Metadata directory not provided
I20260929 16:42:01.657157 6390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:42:01.657429 6390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:42:01.660632 6390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/master-0/data/instance:
uuid: "07195eecc9a34bf195d485c1bf4bd97c"
format_stamp: "Formatted at 2026-09-29 16:42:01 on dist-test-slave-l31f"
server_key: "a18878790a3fbdee5d330589d8b9b4e1"
server_key_iv: "dd1231893dadaa95373b2d139c08ba9f"
server_key_version: "encryptionkey@0"
I20260929 16:42:01.661556 6390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/master-0/wal/instance:
uuid: "07195eecc9a34bf195d485c1bf4bd97c"
format_stamp: "Formatted at 2026-09-29 16:42:01 on dist-test-slave-l31f"
server_key: "a18878790a3fbdee5d330589d8b9b4e1"
server_key_iv: "dd1231893dadaa95373b2d139c08ba9f"
server_key_version: "encryptionkey@0"
I20260929 16:42:01.667512 6390 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260929 16:42:01.671823 6405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:01.673630 6390 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:42:01.673906 6390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "07195eecc9a34bf195d485c1bf4bd97c"
format_stamp: "Formatted at 2026-09-29 16:42:01 on dist-test-slave-l31f"
server_key: "a18878790a3fbdee5d330589d8b9b4e1"
server_key_iv: "dd1231893dadaa95373b2d139c08ba9f"
server_key_version: "encryptionkey@0"
I20260929 16:42:01.674139 6390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:42:01.689203 6390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:42:01.693801 6390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:42:01.694227 6390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:42:01.714741 6390 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.126:42305
I20260929 16:42:01.714843 6456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.126:42305 every 8 connection(s)
I20260929 16:42:01.716285 6390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:42:01.719316 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 6390
I20260929 16:42:01.719559 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/master-0/wal/instance
I20260929 16:42:01.720224 789 external_mini_cluster.cc:1428] Setting key 8ba25253201597c477192fa3f2939ecb
I20260929 16:42:01.720552 6457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 16:42:01 dist-test-slave-l31f krb5kdc[6377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:42:01.737740 6457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07195eecc9a34bf195d485c1bf4bd97c: Bootstrap starting.
I20260929 16:42:01.741427 6457 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07195eecc9a34bf195d485c1bf4bd97c: Neither blocks nor log segments found. Creating new log.
I20260929 16:42:01.742645 6457 log.cc:824] T 00000000000000000000000000000000 P 07195eecc9a34bf195d485c1bf4bd97c: Log is configured to *not* fsync() on all Append() calls
I20260929 16:42:01.745781 6457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07195eecc9a34bf195d485c1bf4bd97c: No bootstrap required, opened a new log
I20260929 16:42:01.746862 6460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:01.722553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35510 (local address 127.0.197.126:42305)
0929 16:42:01.723478 (+ 925us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:01.723491 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:01.723533 (+ 42us) server_negotiation.cc:407] Connection header received
0929 16:42:01.725660 (+ 2127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:01.725685 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:01.726137 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:01.726518 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:01.728585 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:01.730106 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:01.732218 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:01.733029 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:01.733184 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:01.738970 (+ 5786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:01.738979 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:01.738996 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:01.739119 (+ 123us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:01.742413 (+ 3294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:01.743400 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:01.743410 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:01.743417 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:01.743597 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:01.744033 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:01.744037 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:01.744039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:01.744589 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:01.744913 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:01.745330 (+ 417us) server_negotiation.cc:299] Negotiation successful
0929 16:42:01.745784 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":472,"threads_started":1}
I20260929 16:42:01.754472 6457 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07195eecc9a34bf195d485c1bf4bd97c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07195eecc9a34bf195d485c1bf4bd97c" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 42305 } }
I20260929 16:42:01.754947 6457 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07195eecc9a34bf195d485c1bf4bd97c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:42:01.755121 6457 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07195eecc9a34bf195d485c1bf4bd97c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07195eecc9a34bf195d485c1bf4bd97c, State: Initialized, Role: FOLLOWER
I20260929 16:42:01.755808 6457 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07195eecc9a34bf195d485c1bf4bd97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07195eecc9a34bf195d485c1bf4bd97c" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 42305 } }
I20260929 16:42:01.756083 6457 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07195eecc9a34bf195d485c1bf4bd97c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:42:01.756237 6457 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07195eecc9a34bf195d485c1bf4bd97c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:42:01.756419 6457 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07195eecc9a34bf195d485c1bf4bd97c [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:42:01.759040 6457 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07195eecc9a34bf195d485c1bf4bd97c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07195eecc9a34bf195d485c1bf4bd97c" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 42305 } }
I20260929 16:42:01.759771 6457 leader_election.cc:302] T 00000000000000000000000000000000 P 07195eecc9a34bf195d485c1bf4bd97c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07195eecc9a34bf195d485c1bf4bd97c; no voters:
I20260929 16:42:01.760483 6457 leader_election.cc:288] T 00000000000000000000000000000000 P 07195eecc9a34bf195d485c1bf4bd97c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:42:01.760658 6462 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07195eecc9a34bf195d485c1bf4bd97c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:42:01.761538 6462 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07195eecc9a34bf195d485c1bf4bd97c [term 1 LEADER]: Becoming Leader. State: Replica: 07195eecc9a34bf195d485c1bf4bd97c, State: Running, Role: LEADER
I20260929 16:42:01.762106 6462 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07195eecc9a34bf195d485c1bf4bd97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07195eecc9a34bf195d485c1bf4bd97c" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 42305 } }
I20260929 16:42:01.762727 6457 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07195eecc9a34bf195d485c1bf4bd97c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:42:01.765723 6463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07195eecc9a34bf195d485c1bf4bd97c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07195eecc9a34bf195d485c1bf4bd97c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07195eecc9a34bf195d485c1bf4bd97c" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 42305 } } }
I20260929 16:42:01.766456 6463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07195eecc9a34bf195d485c1bf4bd97c [sys.catalog]: This master's current role is: LEADER
I20260929 16:42:01.765700 6464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07195eecc9a34bf195d485c1bf4bd97c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07195eecc9a34bf195d485c1bf4bd97c. Latest consensus state: current_term: 1 leader_uuid: "07195eecc9a34bf195d485c1bf4bd97c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07195eecc9a34bf195d485c1bf4bd97c" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 42305 } } }
I20260929 16:42:01.767767 6464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07195eecc9a34bf195d485c1bf4bd97c [sys.catalog]: This master's current role is: LEADER
I20260929 16:42:01.768615 6471 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:42:01.774385 6471 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:42:01.784996 6471 catalog_manager.cc:1347] Generated new cluster ID: a6187dbc995a4f2c9eeaf275036c3f80
I20260929 16:42:01.785132 6471 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:42:01.824025 6471 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:42:01.826162 6471 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:42:01.838912 6471 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07195eecc9a34bf195d485c1bf4bd97c: Generated new TSK 0
I20260929 16:42:01.840056 6471 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.197.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab.
Entry for principal kudu/127.0.197.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab.
I20260929 16:42:01.906787 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.65:0
--local_ip_for_outbound_sockets=127.0.197.65
--webserver_interface=127.0.197.65
--webserver_port=0
--tserver_master_addrs=127.0.197.126:42305
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab
--principal=kudu/127.0.197.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:35065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:42:02.295958 6483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:42:02.296525 6483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:42:02.297040 6483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:42:02.306617 6483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:42:02.306787 6483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:42:02.307165 6483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.65
I20260929 16:42:02.318734 6483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:35065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab
--principal=kudu/127.0.197.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.197.65
--webserver_port=0
--tserver_master_addrs=127.0.197.126:42305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.65
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:42:02.320676 6483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:42:02.322928 6483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:42:02.340967 6488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:02.341039 6489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:02.340946 6491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:42:02.341746 6483 server_base.cc:1034] running on GCE node
I20260929 16:42:02.342417 6483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:42:02.343735 6483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:42:02.347177 6483 hybrid_clock.cc:648] HybridClock initialized: now 1790700122347104 us; error 64 us; skew 500 ppm
I20260929 16:42:02.347618 6483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:42:02 dist-test-slave-l31f krb5kdc[6377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:42:02.351495 6483 init.cc:377] Logged in from keytab as kudu/127.0.197.65@KRBTEST.COM (short username kudu)
I20260929 16:42:02.353374 6483 webserver.cc:460] Webserver started at http://127.0.197.65:44557/ using document root <none> and password file <none>
I20260929 16:42:02.354190 6483 fs_manager.cc:359] Metadata directory not provided
I20260929 16:42:02.354354 6483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:42:02.354689 6483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:42:02.358029 6483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/ts-0/data/instance:
uuid: "5aacde5ebf814c0fa4c26398d9515827"
format_stamp: "Formatted at 2026-09-29 16:42:02 on dist-test-slave-l31f"
server_key: "1b65f64ba4995a95b261dcd9e0475ecf"
server_key_iv: "983a816fed6b3f3b01bb9bfe379c4e2a"
server_key_version: "encryptionkey@0"
I20260929 16:42:02.358935 6483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/ts-0/wal/instance:
uuid: "5aacde5ebf814c0fa4c26398d9515827"
format_stamp: "Formatted at 2026-09-29 16:42:02 on dist-test-slave-l31f"
server_key: "1b65f64ba4995a95b261dcd9e0475ecf"
server_key_iv: "983a816fed6b3f3b01bb9bfe379c4e2a"
server_key_version: "encryptionkey@0"
I20260929 16:42:02.365471 6483 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 16:42:02.370025 6498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:02.371851 6483 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:42:02.372073 6483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/ts-0/wal
uuid: "5aacde5ebf814c0fa4c26398d9515827"
format_stamp: "Formatted at 2026-09-29 16:42:02 on dist-test-slave-l31f"
server_key: "1b65f64ba4995a95b261dcd9e0475ecf"
server_key_iv: "983a816fed6b3f3b01bb9bfe379c4e2a"
server_key_version: "encryptionkey@0"
I20260929 16:42:02.372314 6483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:42:02.389670 6483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:42:02.394351 6483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:42:02.394686 6483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:42:02.395805 6483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:42:02.397310 6483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:42:02.397428 6483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:02.397607 6483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:42:02.397688 6483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:02.432965 6483 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.65:40399
I20260929 16:42:02.433017 6610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.65:40399 every 8 connection(s)
I20260929 16:42:02.434355 6483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
I20260929 16:42:02.444548 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 6483
I20260929 16:42:02.444778 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/ts-0/wal/instance
I20260929 16:42:02.445402 789 external_mini_cluster.cc:1428] Setting key 314fdc618eb370bf984bf6f3ca6d74e5
Sep 29 16:42:02 dist-test-slave-l31f krb5kdc[6377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.65@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:42:02.461303 6613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:02.437302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:34343 (local address 127.0.197.126:42305)
0929 16:42:02.437780 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:02.437787 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:02.439471 (+ 1684us) server_negotiation.cc:407] Connection header received
0929 16:42:02.441934 (+ 2463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:02.441940 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:02.442024 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:02.442163 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:02.444362 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:02.445242 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:02.447537 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:02.448061 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:02.448182 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:02.452964 (+ 4782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:02.452974 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:02.452982 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:02.453039 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:02.456844 (+ 3805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:02.458065 (+ 1221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:02.458070 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:02.458072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:02.458215 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:02.458799 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:02.458803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:02.458805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:02.459226 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:02.459358 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:02.460485 (+ 1127us) server_negotiation.cc:299] Negotiation successful
0929 16:42:02.460810 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":226,"threads_started":1}
I20260929 16:42:02.462608 6614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:02.437955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:42305 (local address 127.0.197.65:34343)
0929 16:42:02.439226 (+ 1271us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:02.439300 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:02.441486 (+ 2186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:02.442585 (+ 1099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:02.442598 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:02.443375 (+ 777us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:02.444112 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:02.444138 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:02.445446 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:02.445451 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:02.447087 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:02.447127 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:02.448368 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:02.448377 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:02.448751 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:02.449591 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:02.449631 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:02.452605 (+ 2974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:42:02.457081 (+ 4476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:02.457096 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:02.457121 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:02.457898 (+ 777us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:02.458348 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:02.458357 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:02.458363 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:02.458664 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:02.459369 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:42:02.459381 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:42:02.459934 (+ 553us) client_negotiation.cc:769] Sending connection context
0929 16:42:02.460278 (+ 344us) client_negotiation.cc:240] Negotiation successful
0929 16:42:02.460964 (+ 686us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":764,"thread_start_us":450,"threads_started":1}
I20260929 16:42:02.464655 6611 heartbeater.cc:347] Connected to a master server at 127.0.197.126:42305
I20260929 16:42:02.465032 6611 heartbeater.cc:464] Registering TS with master...
I20260929 16:42:02.465965 6611 heartbeater.cc:511] Master 127.0.197.126:42305 requested a full tablet report, sending...
I20260929 16:42:02.469038 6422 ts_manager.cc:194] Registered new tserver with Master: 5aacde5ebf814c0fa4c26398d9515827 (127.0.197.65:40399)
WARNING: no policy specified for kudu/127.0.197.66@KRBTEST.COM; defaulting to no policy
I20260929 16:42:02.470986 6422 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.65@KRBTEST.COM'} at 127.0.197.65:34343
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab.
Entry for principal kudu/127.0.197.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab.
I20260929 16:42:02.504277 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.66:0
--local_ip_for_outbound_sockets=127.0.197.66
--webserver_interface=127.0.197.66
--webserver_port=0
--tserver_master_addrs=127.0.197.126:42305
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab
--principal=kudu/127.0.197.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:35065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:42:02.856673 6617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:42:02.857043 6617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:42:02.857223 6617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:42:02.865068 6617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:42:02.865258 6617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:42:02.865434 6617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.66
I20260929 16:42:02.875855 6617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:35065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab
--principal=kudu/127.0.197.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.197.66
--webserver_port=0
--tserver_master_addrs=127.0.197.126:42305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.66
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:42:02.877357 6617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:42:02.878832 6617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:42:02.892697 6622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:02.892836 6625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:02.892839 6623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:42:02.894374 6617 server_base.cc:1034] running on GCE node
I20260929 16:42:02.895084 6617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:42:02.896175 6617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:42:02.897403 6617 hybrid_clock.cc:648] HybridClock initialized: now 1790700122897337 us; error 61 us; skew 500 ppm
I20260929 16:42:02.897702 6617 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:42:02 dist-test-slave-l31f krb5kdc[6377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:42:02.901278 6617 init.cc:377] Logged in from keytab as kudu/127.0.197.66@KRBTEST.COM (short username kudu)
I20260929 16:42:02.902992 6617 webserver.cc:460] Webserver started at http://127.0.197.66:44761/ using document root <none> and password file <none>
I20260929 16:42:02.903873 6617 fs_manager.cc:359] Metadata directory not provided
I20260929 16:42:02.904022 6617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:42:02.904377 6617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:42:02.907562 6617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/ts-1/data/instance:
uuid: "dabf3b2be28141a0a08adce47f37eb88"
format_stamp: "Formatted at 2026-09-29 16:42:02 on dist-test-slave-l31f"
server_key: "d03631d1b89ce15f4fe3416a22850e8a"
server_key_iv: "9ae640c44471f19b4f4bad6690441e76"
server_key_version: "encryptionkey@0"
I20260929 16:42:02.908406 6617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/ts-1/wal/instance:
uuid: "dabf3b2be28141a0a08adce47f37eb88"
format_stamp: "Formatted at 2026-09-29 16:42:02 on dist-test-slave-l31f"
server_key: "d03631d1b89ce15f4fe3416a22850e8a"
server_key_iv: "9ae640c44471f19b4f4bad6690441e76"
server_key_version: "encryptionkey@0"
I20260929 16:42:02.913992 6617 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:42:02.918200 6632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:02.919869 6617 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:42:02.920077 6617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/ts-1/wal
uuid: "dabf3b2be28141a0a08adce47f37eb88"
format_stamp: "Formatted at 2026-09-29 16:42:02 on dist-test-slave-l31f"
server_key: "d03631d1b89ce15f4fe3416a22850e8a"
server_key_iv: "9ae640c44471f19b4f4bad6690441e76"
server_key_version: "encryptionkey@0"
I20260929 16:42:02.920336 6617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:42:02.936443 6617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:42:02.941138 6617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:42:02.941468 6617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:42:02.942485 6617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:42:02.944094 6617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:42:02.944208 6617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:02.944370 6617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:42:02.944448 6617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:02.981446 6617 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.66:36725
I20260929 16:42:02.981487 6744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.66:36725 every 8 connection(s)
I20260929 16:42:02.982900 6617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
I20260929 16:42:02.988780 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 6617
I20260929 16:42:02.988994 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/ts-1/wal/instance
I20260929 16:42:02.989432 789 external_mini_cluster.cc:1428] Setting key fa1c1bfb92b6cb7565c96b4008af24a0
Sep 29 16:42:02 dist-test-slave-l31f krb5kdc[6377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.66@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:42:03.009296 6747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:02.985929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:54835 (local address 127.0.197.126:42305)
0929 16:42:02.986426 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:02.986431 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:02.987663 (+ 1232us) server_negotiation.cc:407] Connection header received
0929 16:42:02.989691 (+ 2028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:02.989697 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:02.989797 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:02.989925 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:02.991759 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:02.992516 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:02.994155 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:02.994612 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:02.994753 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:03.001010 (+ 6257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:03.001019 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:03.001023 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:03.001102 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:03.005731 (+ 4629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:03.006643 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:03.006650 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:03.006653 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:03.006814 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:03.007394 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:03.007400 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:03.007403 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:03.007698 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:03.007870 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:03.008664 (+ 794us) server_negotiation.cc:299] Negotiation successful
0929 16:42:03.008916 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":267,"threads_started":1}
I20260929 16:42:03.010372 6748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:02.986552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:42305 (local address 127.0.197.66:54835)
0929 16:42:02.987473 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:02.987523 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:02.989362 (+ 1839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:02.990233 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:02.990247 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:02.990845 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:02.991583 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:02.991602 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:02.992693 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:02.992697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:02.993998 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:02.994011 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:02.994866 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:02.994873 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:02.995252 (+ 379us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:02.996602 (+ 1350us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:02.996635 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:03.000589 (+ 3954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:42:03.005939 (+ 5350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:03.005951 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:03.005981 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:03.006483 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:03.006947 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:03.006953 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:03.006957 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:03.007258 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:03.007848 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:42:03.007858 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:42:03.008285 (+ 427us) client_negotiation.cc:769] Sending connection context
0929 16:42:03.008572 (+ 287us) client_negotiation.cc:240] Negotiation successful
0929 16:42:03.009162 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":422,"threads_started":1}
I20260929 16:42:03.012352 6745 heartbeater.cc:347] Connected to a master server at 127.0.197.126:42305
I20260929 16:42:03.012724 6745 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.197.67@KRBTEST.COM; defaulting to no policy
I20260929 16:42:03.013610 6745 heartbeater.cc:511] Master 127.0.197.126:42305 requested a full tablet report, sending...
I20260929 16:42:03.015334 6422 ts_manager.cc:194] Registered new tserver with Master: dabf3b2be28141a0a08adce47f37eb88 (127.0.197.66:36725)
I20260929 16:42:03.016149 6422 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.66@KRBTEST.COM'} at 127.0.197.66:54835
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab.
Entry for principal kudu/127.0.197.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab.
I20260929 16:42:03.047827 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.67:0
--local_ip_for_outbound_sockets=127.0.197.67
--webserver_interface=127.0.197.67
--webserver_port=0
--tserver_master_addrs=127.0.197.126:42305
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab
--principal=kudu/127.0.197.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:35065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:42:03.400390 6751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:42:03.400719 6751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:42:03.400882 6751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:42:03.408952 6751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:42:03.409101 6751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:42:03.409255 6751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.67
I20260929 16:42:03.420033 6751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:35065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab
--principal=kudu/127.0.197.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.197.67
--webserver_port=0
--tserver_master_addrs=127.0.197.126:42305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.67
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:42:03.421495 6751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:42:03.422910 6751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:42:03.437597 6757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:03.437945 6756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:03.437573 6759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:42:03.439098 6751 server_base.cc:1034] running on GCE node
I20260929 16:42:03.439821 6751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:42:03.440883 6751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:42:03.442159 6751 hybrid_clock.cc:648] HybridClock initialized: now 1790700123442112 us; error 46 us; skew 500 ppm
I20260929 16:42:03.442538 6751 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:42:03 dist-test-slave-l31f krb5kdc[6377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:42:03.446091 6751 init.cc:377] Logged in from keytab as kudu/127.0.197.67@KRBTEST.COM (short username kudu)
I20260929 16:42:03.447924 6751 webserver.cc:460] Webserver started at http://127.0.197.67:45389/ using document root <none> and password file <none>
I20260929 16:42:03.448740 6751 fs_manager.cc:359] Metadata directory not provided
I20260929 16:42:03.448917 6751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:42:03.449254 6751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:42:03.453214 6751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/ts-2/data/instance:
uuid: "7ae6605e340f4a7ba8a63b8c269e9d9a"
format_stamp: "Formatted at 2026-09-29 16:42:03 on dist-test-slave-l31f"
server_key: "71295e60201d3dc03711724926f59077"
server_key_iv: "25e474e2c35db8728cd40042fd6402b9"
server_key_version: "encryptionkey@0"
I20260929 16:42:03.454144 6751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/ts-2/wal/instance:
uuid: "7ae6605e340f4a7ba8a63b8c269e9d9a"
format_stamp: "Formatted at 2026-09-29 16:42:03 on dist-test-slave-l31f"
server_key: "71295e60201d3dc03711724926f59077"
server_key_iv: "25e474e2c35db8728cd40042fd6402b9"
server_key_version: "encryptionkey@0"
I20260929 16:42:03.460734 6751 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260929 16:42:03.465083 6766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:03.466820 6751 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260929 16:42:03.467108 6751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/ts-2/wal
uuid: "7ae6605e340f4a7ba8a63b8c269e9d9a"
format_stamp: "Formatted at 2026-09-29 16:42:03 on dist-test-slave-l31f"
server_key: "71295e60201d3dc03711724926f59077"
server_key_iv: "25e474e2c35db8728cd40042fd6402b9"
server_key_version: "encryptionkey@0"
I20260929 16:42:03.467383 6751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:42:03.475240 6611 heartbeater.cc:503] Master 127.0.197.126:42305 was elected leader, sending a full tablet report...
I20260929 16:42:03.486122 6751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:42:03.490718 6751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:42:03.490995 6751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:42:03.492062 6751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:42:03.493418 6751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:42:03.493507 6751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:03.493594 6751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:42:03.493651 6751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:03.532351 6751 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.67:34873
I20260929 16:42:03.532397 6878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.67:34873 every 8 connection(s)
I20260929 16:42:03.533844 6751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
I20260929 16:42:03.542413 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 6751
I20260929 16:42:03.542640 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/minicluster-data/ts-2/wal/instance
I20260929 16:42:03.543138 789 external_mini_cluster.cc:1428] Setting key 5b03744a0a3717ea1d3b58630cdfba5d
Sep 29 16:42:03 dist-test-slave-l31f krb5kdc[6377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.67@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:42:03.556270 6881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:03.536749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:46853 (local address 127.0.197.126:42305)
0929 16:42:03.537240 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:03.537245 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:03.538163 (+ 918us) server_negotiation.cc:407] Connection header received
0929 16:42:03.540132 (+ 1969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:03.540138 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:03.540238 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:03.540373 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:03.542007 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:03.542911 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:03.544701 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:03.545148 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:03.545306 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:03.549951 (+ 4645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:03.549957 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:03.549960 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:03.549998 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:03.552889 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:03.553791 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:03.553797 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:03.553800 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:03.553923 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:03.554399 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:03.554403 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:03.554406 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:03.554671 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:03.554840 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:03.555633 (+ 793us) server_negotiation.cc:299] Negotiation successful
0929 16:42:03.555876 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":238,"threads_started":1}
I20260929 16:42:03.557310 6882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:03.537213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:42305 (local address 127.0.197.67:46853)
0929 16:42:03.537996 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:03.538036 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:03.539813 (+ 1777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:03.540633 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:03.540642 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:03.541200 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:03.541816 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:03.541836 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:03.543135 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:03.543139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:03.544530 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:03.544545 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:03.545398 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:03.545408 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:03.545665 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:03.546409 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:03.546432 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:03.549649 (+ 3217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:42:03.553088 (+ 3439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:03.553104 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:03.553121 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:03.553638 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:03.554050 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:03.554056 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:03.554064 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:03.554257 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:03.554806 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:42:03.554815 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:42:03.555248 (+ 433us) client_negotiation.cc:769] Sending connection context
0929 16:42:03.555527 (+ 279us) client_negotiation.cc:240] Negotiation successful
0929 16:42:03.556100 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":308,"threads_started":1}
I20260929 16:42:03.559037 6879 heartbeater.cc:347] Connected to a master server at 127.0.197.126:42305
I20260929 16:42:03.559389 6879 heartbeater.cc:464] Registering TS with master...
I20260929 16:42:03.560186 6879 heartbeater.cc:511] Master 127.0.197.126:42305 requested a full tablet report, sending...
I20260929 16:42:03.561757 6422 ts_manager.cc:194] Registered new tserver with Master: 7ae6605e340f4a7ba8a63b8c269e9d9a (127.0.197.67:34873)
I20260929 16:42:03.562659 6422 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.67@KRBTEST.COM'} at 127.0.197.67:46853
I20260929 16:42:03.571702 789 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/krb5kdc/test-admin.keytab.
Sep 29 16:42:03 dist-test-slave-l31f krb5kdc[6377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:42:03.602205 789 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260929 16:42:03.607714 6887 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:42:03.605069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44362 (local address 127.0.197.65:40399)
0929 16:42:03.605743 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:03.605751 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:03.605782 (+ 31us) server_negotiation.cc:407] Connection header received
0929 16:42:03.605853 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:03.605859 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:03.606082 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:03.606216 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:03.607306 (+ 1090us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44362: BlockingRecv error: recv got EOF from 127.0.0.1:44362 (error 108)
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":346,"threads_started":1}
W20260929 16:42:03.607843 789 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/krb5kdc/krb5cc' of type FILE
Sep 29 16:42:03 dist-test-slave-l31f krb5kdc[6377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:42:03.609566 789 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 16:42:03 dist-test-slave-l31f krb5kdc[6377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:42:03.611001 789 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 16:42:03 dist-test-slave-l31f krb5kdc[6377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.65@KRBTEST.COM
I20260929 16:42:03.628139 6887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:03.612796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44378 (local address 127.0.197.65:40399)
0929 16:42:03.613029 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:03.613036 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:03.613220 (+ 184us) server_negotiation.cc:407] Connection header received
0929 16:42:03.613386 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:03.613390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:03.613454 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:03.613542 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:03.614532 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:03.615620 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:03.617054 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:03.617709 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:03.617825 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:03.621856 (+ 4031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:03.621876 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:03.621893 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:03.621945 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:03.625241 (+ 3296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:03.626035 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:03.626042 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:03.626044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:03.626171 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:03.626551 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:03.626554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:03.626556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:03.626930 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:03.627227 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:03.627559 (+ 332us) server_negotiation.cc:299] Negotiation successful
0929 16:42:03.627820 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260929 16:42:03.634516 6887 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:42:03.632635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44384 (local address 127.0.197.65:40399)
0929 16:42:03.632881 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:03.632888 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:03.633057 (+ 169us) server_negotiation.cc:407] Connection header received
0929 16:42:03.633271 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:03.633278 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:03.633360 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:03.633473 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:03.634237 (+ 764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44384: BlockingRecv error: recv got EOF from 127.0.0.1:44384 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260929 16:42:03.634707 789 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/krb5kdc/krb5cc' of type FILE
Sep 29 16:42:03 dist-test-slave-l31f krb5kdc[6377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:42:03.636271 789 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 16:42:03 dist-test-slave-l31f krb5kdc[6377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:42:03.638000 789 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 16:42:03 dist-test-slave-l31f krb5kdc[6377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.65@KRBTEST.COM
I20260929 16:42:03.664546 6887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:03.640031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44388 (local address 127.0.197.65:40399)
0929 16:42:03.640203 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:03.640207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:03.640371 (+ 164us) server_negotiation.cc:407] Connection header received
0929 16:42:03.640545 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:03.640549 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:03.640641 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:03.640731 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:03.642342 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:03.643212 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:03.644542 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:03.644989 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:03.645090 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:03.658155 (+ 13065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:03.658162 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:03.658166 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:03.658214 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:03.661479 (+ 3265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:03.662315 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:03.662319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:03.662322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:03.662458 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:03.662999 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:03.663050 (+ 51us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:03.663058 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:03.663404 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:03.663623 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:03.664057 (+ 434us) server_negotiation.cc:299] Negotiation successful
0929 16:42:03.664238 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260929 16:42:03.669708 6887 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:42:03.667829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44392 (local address 127.0.197.65:40399)
0929 16:42:03.668051 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:03.668056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:03.668234 (+ 178us) server_negotiation.cc:407] Connection header received
0929 16:42:03.668406 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:03.668413 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:03.668489 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:03.668571 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:03.669454 (+ 883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44392: BlockingRecv error: recv got EOF from 127.0.0.1:44392 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260929 16:42:03.669960 789 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790700068255611-789-0/krb5kdc/krb5cc' of type FILE
Sep 29 16:42:03 dist-test-slave-l31f krb5kdc[6377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:42:03.671576 789 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 16:42:03 dist-test-slave-l31f krb5kdc[6377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:42:03.672904 789 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 16:42:03 dist-test-slave-l31f krb5kdc[6377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.65@KRBTEST.COM
I20260929 16:42:03.689431 6887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:03.674321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44394 (local address 127.0.197.65:40399)
0929 16:42:03.674561 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:03.674567 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:03.674711 (+ 144us) server_negotiation.cc:407] Connection header received
0929 16:42:03.675100 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:03.675104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:03.675169 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:03.675261 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:03.676312 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:03.677158 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:03.678681 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:03.679293 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:03.679405 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:03.683909 (+ 4504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:03.683915 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:03.683918 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:03.683977 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:03.686562 (+ 2585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:03.687441 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:03.687446 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:03.687448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:03.687612 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:03.688055 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:03.688059 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:03.688062 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:03.688338 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:03.688596 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:03.688870 (+ 274us) server_negotiation.cc:299] Negotiation successful
0929 16:42:03.689041 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 16:42:03.691057 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 6483
I20260929 16:42:03.704334 6887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:03.691926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44396 (local address 127.0.197.65:40399)
0929 16:42:03.692246 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:03.692252 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:03.692472 (+ 220us) server_negotiation.cc:407] Connection header received
0929 16:42:03.692603 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:03.692607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:03.692666 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:03.692745 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:03.693700 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:03.694372 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:03.695708 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:03.696154 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:03.696253 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:03.698409 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:03.698413 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:03.698417 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:03.698447 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:03.701435 (+ 2988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:03.702218 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:03.702223 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:03.702225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:03.702368 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:03.702839 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:03.702843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:03.702845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:03.703214 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:03.703441 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:03.703796 (+ 355us) server_negotiation.cc:299] Negotiation successful
0929 16:42:03.704061 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162}
I20260929 16:42:03.705076 6606 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:42:03.790498 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 6483
I20260929 16:42:03.808063 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 6617
Sep 29 16:42:03 dist-test-slave-l31f krb5kdc[6377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.66@KRBTEST.COM
I20260929 16:42:03.825084 6900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:03.808901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43342 (local address 127.0.197.66:36725)
0929 16:42:03.809396 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:03.809403 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:03.809490 (+ 87us) server_negotiation.cc:407] Connection header received
0929 16:42:03.809726 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:03.809735 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:03.809973 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:03.810110 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:03.811303 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:03.812316 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:03.813599 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:03.814224 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:03.814342 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:03.818771 (+ 4429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:03.818787 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:03.818804 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:03.818857 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:03.821911 (+ 3054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:03.822785 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:03.822794 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:03.822796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:03.822938 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:03.823374 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:03.823378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:03.823380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:03.823841 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:03.824089 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:03.824459 (+ 370us) server_negotiation.cc:299] Negotiation successful
0929 16:42:03.824718 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":225,"threads_started":1}
I20260929 16:42:03.826440 6740 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:42:03.913549 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 6617
I20260929 16:42:03.930776 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 6751
Sep 29 16:42:03 dist-test-slave-l31f krb5kdc[6377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.67@KRBTEST.COM
I20260929 16:42:03.948390 6902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:03.932022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59720 (local address 127.0.197.67:34873)
0929 16:42:03.932728 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:03.932734 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:03.932766 (+ 32us) server_negotiation.cc:407] Connection header received
0929 16:42:03.932837 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:03.932844 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:03.933059 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:03.933201 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:03.934357 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:03.935546 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:03.936954 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:03.937548 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:03.937670 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:03.941973 (+ 4303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:03.941988 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:03.942005 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:03.942047 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:03.945281 (+ 3234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:03.946106 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:03.946114 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:03.946116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:03.946261 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:03.946739 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:03.946743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:03.946745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:03.947234 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:03.947489 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:03.947765 (+ 276us) server_negotiation.cc:299] Negotiation successful
0929 16:42:03.948038 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":390,"threads_started":1}
I20260929 16:42:03.949812 6874 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:42:04.035219 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 6751
I20260929 16:42:04.052098 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 6390
I20260929 16:42:04.053328 6452 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:42:04.159921 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 6390
2026-09-29T16:42:04Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4265 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:42:04 dist-test-slave-l31f krb5kdc[6907](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:42:04 dist-test-slave-l31f krb5kdc[6907](info): set up 2 sockets
Sep 29 16:42:04 dist-test-slave-l31f krb5kdc[6907](info): commencing operation
krb5kdc: starting...
W20260929 16:42:05.280045 789 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/test-user.keytab.
Sep 29 16:42:05 dist-test-slave-l31f krb5kdc[6907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:42:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:42:05Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.197.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.197.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.197.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/oryx_127.0.197.126.keytab.
Entry for principal oryx/127.0.197.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/oryx_127.0.197.126.keytab.
I20260929 16:42:05.519976 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:33747
--webserver_interface=127.0.197.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/oryx_127.0.197.126.keytab
--principal=oryx/127.0.197.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:34055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.126:33747
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:42:05.885042 6920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:42:05.885426 6920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:42:05.885593 6920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:42:05.893654 6920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:42:05.893832 6920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:42:05.893918 6920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:42:05.894032 6920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:42:05.894101 6920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:42:05.904559 6920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:34055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.197.126:33747
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/oryx_127.0.197.126.keytab
--principal=oryx/127.0.197.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:33747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.197.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:42:05.906086 6920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:42:05.907581 6920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:42:05.922174 6925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:05.922174 6926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:05.922873 6928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:42:05.923094 6920 server_base.cc:1034] running on GCE node
I20260929 16:42:05.923997 6920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:42:05.925478 6920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:42:05.926697 6920 hybrid_clock.cc:648] HybridClock initialized: now 1790700125926637 us; error 67 us; skew 500 ppm
I20260929 16:42:05.927145 6920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:42:05 dist-test-slave-l31f krb5kdc[6907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700125, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.197.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:42:05.930655 6920 init.cc:377] Logged in from keytab as oryx/127.0.197.126@KRBTEST.COM (short username oryx)
I20260929 16:42:05.932520 6920 webserver.cc:460] Webserver started at http://127.0.197.126:33973/ using document root <none> and password file <none>
I20260929 16:42:05.933324 6920 fs_manager.cc:359] Metadata directory not provided
I20260929 16:42:05.933477 6920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:42:05.933836 6920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:42:05.937296 6920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-0/data/instance:
uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f"
format_stamp: "Formatted at 2026-09-29 16:42:05 on dist-test-slave-l31f"
server_key: "51d08f7161d3afee79c4bbbabc5a2f68"
server_key_iv: "6442b5bb89be251e42a3444c1890c93d"
server_key_version: "encryptionkey@0"
I20260929 16:42:05.938242 6920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-0/wal/instance:
uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f"
format_stamp: "Formatted at 2026-09-29 16:42:05 on dist-test-slave-l31f"
server_key: "51d08f7161d3afee79c4bbbabc5a2f68"
server_key_iv: "6442b5bb89be251e42a3444c1890c93d"
server_key_version: "encryptionkey@0"
I20260929 16:42:05.944293 6920 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 16:42:05.948318 6935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:05.949980 6920 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:42:05.950219 6920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f"
format_stamp: "Formatted at 2026-09-29 16:42:05 on dist-test-slave-l31f"
server_key: "51d08f7161d3afee79c4bbbabc5a2f68"
server_key_iv: "6442b5bb89be251e42a3444c1890c93d"
server_key_version: "encryptionkey@0"
I20260929 16:42:05.950449 6920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:42:05.967940 6920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:42:05.972595 6920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:42:05.972916 6920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:42:05.992617 6920 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.126:33747
I20260929 16:42:05.992648 6986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.126:33747 every 8 connection(s)
I20260929 16:42:05.994115 6920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:42:05.998270 6987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:42:06.000042 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 6920
I20260929 16:42:06.000293 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-0/wal/instance
I20260929 16:42:06.001120 789 external_mini_cluster.cc:1428] Setting key 7bfaa55b4bf985c453ee919096700542
I20260929 16:42:06.013512 6987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f: Bootstrap starting.
Sep 29 16:42:06 dist-test-slave-l31f krb5kdc[6907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM
I20260929 16:42:06.017202 6987 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f: Neither blocks nor log segments found. Creating new log.
I20260929 16:42:06.018649 6987 log.cc:824] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f: Log is configured to *not* fsync() on all Append() calls
I20260929 16:42:06.021988 6987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f: No bootstrap required, opened a new log
I20260929 16:42:06.024893 6990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:06.003505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46088 (local address 127.0.197.126:33747)
0929 16:42:06.004424 (+ 919us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:06.004437 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:06.004480 (+ 43us) server_negotiation.cc:407] Connection header received
0929 16:42:06.006133 (+ 1653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:06.006161 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:06.006627 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:06.007083 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:06.008594 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:06.009985 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:06.011438 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:06.012206 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:06.012420 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:06.016790 (+ 4370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:06.016800 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:06.016817 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:06.016864 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:06.020526 (+ 3662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:06.021320 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:06.021327 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:06.021334 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:06.021506 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:06.021904 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:06.021908 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:06.021910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:06.022504 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:06.022840 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:06.023304 (+ 464us) server_negotiation.cc:299] Negotiation successful
0929 16:42:06.023778 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":370,"threads_started":1}
I20260929 16:42:06.031711 6987 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33747 } }
I20260929 16:42:06.032296 6987 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:42:06.032439 6987 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1f8d56bd194fb2b0c5bd8cc6c2993f, State: Initialized, Role: FOLLOWER
I20260929 16:42:06.033159 6987 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33747 } }
I20260929 16:42:06.033422 6987 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:42:06.033560 6987 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:42:06.033754 6987 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:42:06.036214 6987 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33747 } }
I20260929 16:42:06.036878 6987 leader_election.cc:302] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab1f8d56bd194fb2b0c5bd8cc6c2993f; no voters:
I20260929 16:42:06.037712 6987 leader_election.cc:288] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:42:06.037814 6992 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:42:06.038518 6992 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f [term 1 LEADER]: Becoming Leader. State: Replica: ab1f8d56bd194fb2b0c5bd8cc6c2993f, State: Running, Role: LEADER
I20260929 16:42:06.039238 6992 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33747 } }
I20260929 16:42:06.040365 6987 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:42:06.042783 6994 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab1f8d56bd194fb2b0c5bd8cc6c2993f. Latest consensus state: current_term: 1 leader_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33747 } } }
I20260929 16:42:06.043238 6994 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f [sys.catalog]: This master's current role is: LEADER
I20260929 16:42:06.044083 6993 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33747 } } }
I20260929 16:42:06.044443 6993 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f [sys.catalog]: This master's current role is: LEADER
I20260929 16:42:06.045696 7000 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:42:06.050817 7000 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:42:06.060631 7000 catalog_manager.cc:1347] Generated new cluster ID: f0d8d655a3754a4e8a727c9f26d8474b
I20260929 16:42:06.060782 7000 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:42:06.071762 7000 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:42:06.073794 7000 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:42:06.083135 7000 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f: Generated new TSK 0
I20260929 16:42:06.084383 7000 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.197.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.197.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.197.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/oryx_127.0.197.65.keytab.
Entry for principal oryx/127.0.197.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/oryx_127.0.197.65.keytab.
I20260929 16:42:06.144148 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.65:0
--local_ip_for_outbound_sockets=127.0.197.65
--webserver_interface=127.0.197.65
--webserver_port=0
--tserver_master_addrs=127.0.197.126:33747
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/oryx_127.0.197.65.keytab
--principal=oryx/127.0.197.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:34055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:42:06.512401 7013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:42:06.512795 7013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:42:06.512969 7013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:42:06.520768 7013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:42:06.520944 7013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:42:06.521128 7013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.65
I20260929 16:42:06.532915 7013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:34055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/oryx_127.0.197.65.keytab
--principal=oryx/127.0.197.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.197.65
--webserver_port=0
--tserver_master_addrs=127.0.197.126:33747
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.65
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:42:06.534389 7013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:42:06.535950 7013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:42:06.552954 7019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:06.552997 7018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:06.553195 7021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:42:06.554076 7013 server_base.cc:1034] running on GCE node
I20260929 16:42:06.554943 7013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:42:06.556231 7013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:42:06.557713 7013 hybrid_clock.cc:648] HybridClock initialized: now 1790700126557618 us; error 49 us; skew 500 ppm
I20260929 16:42:06.558446 7013 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:42:06 dist-test-slave-l31f krb5kdc[6907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700126, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.197.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:42:06.563556 7013 init.cc:377] Logged in from keytab as oryx/127.0.197.65@KRBTEST.COM (short username oryx)
I20260929 16:42:06.566186 7013 webserver.cc:460] Webserver started at http://127.0.197.65:40677/ using document root <none> and password file <none>
I20260929 16:42:06.567422 7013 fs_manager.cc:359] Metadata directory not provided
I20260929 16:42:06.567637 7013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:42:06.568075 7013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:42:06.572367 7013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-0/data/instance:
uuid: "52819fca8649475f98ef4c458a2e9a43"
format_stamp: "Formatted at 2026-09-29 16:42:06 on dist-test-slave-l31f"
server_key: "9ccc0bf0d0c14f8ae66feb3cdc130f28"
server_key_iv: "c3b9bf2ee5f60f4b4f1abe75f5d6390a"
server_key_version: "encryptionkey@0"
I20260929 16:42:06.573356 7013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-0/wal/instance:
uuid: "52819fca8649475f98ef4c458a2e9a43"
format_stamp: "Formatted at 2026-09-29 16:42:06 on dist-test-slave-l31f"
server_key: "9ccc0bf0d0c14f8ae66feb3cdc130f28"
server_key_iv: "c3b9bf2ee5f60f4b4f1abe75f5d6390a"
server_key_version: "encryptionkey@0"
I20260929 16:42:06.580936 7013 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.003s
I20260929 16:42:06.586253 7028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:06.588919 7013 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260929 16:42:06.589291 7013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-0/wal
uuid: "52819fca8649475f98ef4c458a2e9a43"
format_stamp: "Formatted at 2026-09-29 16:42:06 on dist-test-slave-l31f"
server_key: "9ccc0bf0d0c14f8ae66feb3cdc130f28"
server_key_iv: "c3b9bf2ee5f60f4b4f1abe75f5d6390a"
server_key_version: "encryptionkey@0"
I20260929 16:42:06.589818 7013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:42:06.611135 7013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:42:06.617044 7013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:42:06.617547 7013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:42:06.618693 7013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:42:06.620334 7013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:42:06.620445 7013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:06.620615 7013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:42:06.620688 7013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:06.657958 7013 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.65:35927
I20260929 16:42:06.658003 7140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.65:35927 every 8 connection(s)
I20260929 16:42:06.659478 7013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
I20260929 16:42:06.660938 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 7013
I20260929 16:42:06.661139 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-0/wal/instance
I20260929 16:42:06.661964 789 external_mini_cluster.cc:1428] Setting key b6e621dafaeb65a0cc45c116f6392502
Sep 29 16:42:06 dist-test-slave-l31f krb5kdc[6907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700126, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.197.65@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM
I20260929 16:42:06.685221 7143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:06.662648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:38103 (local address 127.0.197.126:33747)
0929 16:42:06.663430 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:06.663435 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:06.664607 (+ 1172us) server_negotiation.cc:407] Connection header received
0929 16:42:06.666553 (+ 1946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:06.666561 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:06.666653 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:06.666788 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:06.668591 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:06.669780 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:06.672317 (+ 2537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:06.672737 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:06.672853 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:06.677493 (+ 4640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:06.677503 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:06.677511 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:06.677563 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:06.681387 (+ 3824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:06.682345 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:06.682354 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:06.682360 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:06.682536 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:06.683087 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:06.683096 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:06.683103 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:06.683512 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:06.683711 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:06.684490 (+ 779us) server_negotiation.cc:299] Negotiation successful
0929 16:42:06.684744 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":303,"threads_started":1}
WARNING: no policy specified for oryx/127.0.197.66@KRBTEST.COM; defaulting to no policy
I20260929 16:42:06.686015 7144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:06.663508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:33747 (local address 127.0.197.65:38103)
0929 16:42:06.664431 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:06.664488 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:06.666244 (+ 1756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:06.667135 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:06.667151 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:06.667717 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:06.668387 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:06.668408 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:06.669983 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:06.669987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:06.672098 (+ 2111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:06.672120 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:06.673005 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:06.673015 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:06.673387 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:06.674224 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:06.674269 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:06.677170 (+ 2901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:42:06.681629 (+ 4459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:06.681639 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:06.681656 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:06.682168 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:06.682687 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:06.682690 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:06.682692 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:06.682913 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:06.683691 (+ 778us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:42:06.683703 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:42:06.684069 (+ 366us) client_negotiation.cc:769] Sending connection context
0929 16:42:06.684316 (+ 247us) client_negotiation.cc:240] Negotiation successful
0929 16:42:06.684745 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":384,"threads_started":1}
I20260929 16:42:06.687860 7141 heartbeater.cc:347] Connected to a master server at 127.0.197.126:33747
I20260929 16:42:06.688222 7141 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.197.66@KRBTEST.COM" created.
I20260929 16:42:06.689170 7141 heartbeater.cc:511] Master 127.0.197.126:33747 requested a full tablet report, sending...
I20260929 16:42:06.691684 6952 ts_manager.cc:194] Registered new tserver with Master: 52819fca8649475f98ef4c458a2e9a43 (127.0.197.65:35927)
I20260929 16:42:06.693233 6952 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.197.65@KRBTEST.COM'} at 127.0.197.65:38103
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.197.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/oryx_127.0.197.66.keytab.
Entry for principal oryx/127.0.197.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/oryx_127.0.197.66.keytab.
I20260929 16:42:06.718564 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.66:0
--local_ip_for_outbound_sockets=127.0.197.66
--webserver_interface=127.0.197.66
--webserver_port=0
--tserver_master_addrs=127.0.197.126:33747
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/oryx_127.0.197.66.keytab
--principal=oryx/127.0.197.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:34055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:42:07.083101 7147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:42:07.083426 7147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:42:07.083582 7147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:42:07.091415 7147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:42:07.091543 7147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:42:07.091739 7147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.66
I20260929 16:42:07.102324 7147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:34055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/oryx_127.0.197.66.keytab
--principal=oryx/127.0.197.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.197.66
--webserver_port=0
--tserver_master_addrs=127.0.197.126:33747
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.66
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:42:07.103865 7147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:42:07.105252 7147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:42:07.120024 7152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:07.120119 7153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:07.120297 7155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:42:07.121742 7147 server_base.cc:1034] running on GCE node
I20260929 16:42:07.122449 7147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:42:07.123785 7147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:42:07.125083 7147 hybrid_clock.cc:648] HybridClock initialized: now 1790700127125015 us; error 54 us; skew 500 ppm
I20260929 16:42:07.125487 7147 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:42:07 dist-test-slave-l31f krb5kdc[6907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700127, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.197.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:42:07.129034 7147 init.cc:377] Logged in from keytab as oryx/127.0.197.66@KRBTEST.COM (short username oryx)
I20260929 16:42:07.130950 7147 webserver.cc:460] Webserver started at http://127.0.197.66:44309/ using document root <none> and password file <none>
I20260929 16:42:07.131844 7147 fs_manager.cc:359] Metadata directory not provided
I20260929 16:42:07.132004 7147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:42:07.132334 7147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:42:07.135599 7147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-1/data/instance:
uuid: "9bed1bd04d834cdd9d75106656df7153"
format_stamp: "Formatted at 2026-09-29 16:42:07 on dist-test-slave-l31f"
server_key: "6c5b41f72cba70e07cc0aad0e22a1405"
server_key_iv: "dbaf7929c03853ab73005d00c0b6cc0c"
server_key_version: "encryptionkey@0"
I20260929 16:42:07.136477 7147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-1/wal/instance:
uuid: "9bed1bd04d834cdd9d75106656df7153"
format_stamp: "Formatted at 2026-09-29 16:42:07 on dist-test-slave-l31f"
server_key: "6c5b41f72cba70e07cc0aad0e22a1405"
server_key_iv: "dbaf7929c03853ab73005d00c0b6cc0c"
server_key_version: "encryptionkey@0"
I20260929 16:42:07.142573 7147 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 16:42:07.147161 7162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:07.148952 7147 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260929 16:42:07.149188 7147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-1/wal
uuid: "9bed1bd04d834cdd9d75106656df7153"
format_stamp: "Formatted at 2026-09-29 16:42:07 on dist-test-slave-l31f"
server_key: "6c5b41f72cba70e07cc0aad0e22a1405"
server_key_iv: "dbaf7929c03853ab73005d00c0b6cc0c"
server_key_version: "encryptionkey@0"
I20260929 16:42:07.149435 7147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:42:07.180179 7147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:42:07.185048 7147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:42:07.185411 7147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:42:07.186741 7147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:42:07.188272 7147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:42:07.188355 7147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:07.188499 7147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:42:07.188558 7147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:07.225962 7147 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.66:44381
I20260929 16:42:07.226008 7274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.66:44381 every 8 connection(s)
I20260929 16:42:07.227378 7147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
I20260929 16:42:07.232793 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 7147
I20260929 16:42:07.232980 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-1/wal/instance
I20260929 16:42:07.233516 789 external_mini_cluster.cc:1428] Setting key 46716bdd06905aca56ea80fac8003e2f
Sep 29 16:42:07 dist-test-slave-l31f krb5kdc[6907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700127, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.197.66@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM
I20260929 16:42:07.249480 7277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:07.230165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:39811 (local address 127.0.197.126:33747)
0929 16:42:07.230622 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:07.230627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:07.231544 (+ 917us) server_negotiation.cc:407] Connection header received
0929 16:42:07.233567 (+ 2023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:07.233573 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:07.233674 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:07.233806 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:07.235501 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:07.236253 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:07.237889 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:07.238324 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:07.238469 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:07.243134 (+ 4665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:07.243139 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:07.243142 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:07.243187 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:07.246042 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:07.246918 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:07.246924 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:07.246926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:07.247136 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:07.247607 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:07.247613 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:07.247616 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:07.247908 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:07.248071 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:07.248828 (+ 757us) server_negotiation.cc:299] Negotiation successful
0929 16:42:07.249073 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":249,"threads_started":1}
I20260929 16:42:07.250597 7278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:07.230613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:33747 (local address 127.0.197.66:39811)
0929 16:42:07.231380 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:07.231422 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:07.233263 (+ 1841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:07.234069 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:07.234078 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:07.234618 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:07.235319 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:07.235341 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:07.236430 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:07.236434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:07.237728 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:07.237741 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:07.238567 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:07.238574 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:07.238828 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:07.239773 (+ 945us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:07.239810 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:07.242792 (+ 2982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:42:07.246239 (+ 3447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:07.246251 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:07.246279 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:07.246762 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:07.247263 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:07.247268 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:07.247272 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:07.247479 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:07.248046 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:42:07.248055 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:42:07.248446 (+ 391us) client_negotiation.cc:769] Sending connection context
0929 16:42:07.248738 (+ 292us) client_negotiation.cc:240] Negotiation successful
0929 16:42:07.249318 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":348,"threads_started":1}
I20260929 16:42:07.252398 7275 heartbeater.cc:347] Connected to a master server at 127.0.197.126:33747
I20260929 16:42:07.252745 7275 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for oryx/127.0.197.67@KRBTEST.COM; defaulting to no policy
I20260929 16:42:07.253557 7275 heartbeater.cc:511] Master 127.0.197.126:33747 requested a full tablet report, sending...
I20260929 16:42:07.255450 6952 ts_manager.cc:194] Registered new tserver with Master: 9bed1bd04d834cdd9d75106656df7153 (127.0.197.66:44381)
I20260929 16:42:07.256383 6952 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.197.66@KRBTEST.COM'} at 127.0.197.66:39811
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.197.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.197.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/oryx_127.0.197.67.keytab.
Entry for principal oryx/127.0.197.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/oryx_127.0.197.67.keytab.
I20260929 16:42:07.283480 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.67:0
--local_ip_for_outbound_sockets=127.0.197.67
--webserver_interface=127.0.197.67
--webserver_port=0
--tserver_master_addrs=127.0.197.126:33747
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/oryx_127.0.197.67.keytab
--principal=oryx/127.0.197.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:34055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:42:07.631896 7281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:42:07.632212 7281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:42:07.632372 7281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:42:07.640026 7281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:42:07.640249 7281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:42:07.640452 7281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.67
I20260929 16:42:07.652115 7281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:34055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/oryx_127.0.197.67.keytab
--principal=oryx/127.0.197.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.197.67
--webserver_port=0
--tserver_master_addrs=127.0.197.126:33747
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.67
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:42:07.653635 7281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:42:07.655239 7281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:42:07.669695 7287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:07.669754 7286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:07.669711 7289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:42:07.671181 7281 server_base.cc:1034] running on GCE node
I20260929 16:42:07.671909 7281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:42:07.672967 7281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:42:07.674182 7281 hybrid_clock.cc:648] HybridClock initialized: now 1790700127674138 us; error 38 us; skew 500 ppm
I20260929 16:42:07.674537 7281 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:42:07 dist-test-slave-l31f krb5kdc[6907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700127, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.197.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:42:07.678092 7281 init.cc:377] Logged in from keytab as oryx/127.0.197.67@KRBTEST.COM (short username oryx)
I20260929 16:42:07.679852 7281 webserver.cc:460] Webserver started at http://127.0.197.67:42887/ using document root <none> and password file <none>
I20260929 16:42:07.680603 7281 fs_manager.cc:359] Metadata directory not provided
I20260929 16:42:07.680769 7281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:42:07.681092 7281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:42:07.684350 7281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-2/data/instance:
uuid: "af179336d9f44d9ebe6d2205959274e4"
format_stamp: "Formatted at 2026-09-29 16:42:07 on dist-test-slave-l31f"
server_key: "102cba67c703b9db4b69cd2e9e617615"
server_key_iv: "0411ff8c1c934eb05d0b12bec7bca09f"
server_key_version: "encryptionkey@0"
I20260929 16:42:07.685268 7281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-2/wal/instance:
uuid: "af179336d9f44d9ebe6d2205959274e4"
format_stamp: "Formatted at 2026-09-29 16:42:07 on dist-test-slave-l31f"
server_key: "102cba67c703b9db4b69cd2e9e617615"
server_key_iv: "0411ff8c1c934eb05d0b12bec7bca09f"
server_key_version: "encryptionkey@0"
I20260929 16:42:07.691339 7281 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260929 16:42:07.695613 7296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:07.697188 7281 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 16:42:07.697490 7141 heartbeater.cc:503] Master 127.0.197.126:33747 was elected leader, sending a full tablet report...
I20260929 16:42:07.697410 7281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-2/wal
uuid: "af179336d9f44d9ebe6d2205959274e4"
format_stamp: "Formatted at 2026-09-29 16:42:07 on dist-test-slave-l31f"
server_key: "102cba67c703b9db4b69cd2e9e617615"
server_key_iv: "0411ff8c1c934eb05d0b12bec7bca09f"
server_key_version: "encryptionkey@0"
I20260929 16:42:07.697635 7281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:42:07.734549 7281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:42:07.739483 7281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:42:07.739785 7281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:42:07.740897 7281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:42:07.742494 7281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:42:07.742574 7281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:07.742668 7281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:42:07.742723 7281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:07.777712 7281 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.67:36295
I20260929 16:42:07.777813 7408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.67:36295 every 8 connection(s)
I20260929 16:42:07.779254 7281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
I20260929 16:42:07.779855 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 7281
I20260929 16:42:07.780045 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-2/wal/instance
I20260929 16:42:07.780676 789 external_mini_cluster.cc:1428] Setting key 3a06904ded2993f16143e704b44b5c3f
Sep 29 16:42:07 dist-test-slave-l31f krb5kdc[6907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700127, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.197.67@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM
I20260929 16:42:07.803263 7411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:07.782322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:47083 (local address 127.0.197.126:33747)
0929 16:42:07.782812 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:07.782819 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:07.784649 (+ 1830us) server_negotiation.cc:407] Connection header received
0929 16:42:07.786745 (+ 2096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:07.786778 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:07.786856 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:07.786975 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:07.789072 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:07.790067 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:07.792089 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:07.792576 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:07.792724 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:07.797350 (+ 4626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:07.797355 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:07.797359 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:07.797396 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:07.799917 (+ 2521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:07.800761 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:07.800766 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:07.800769 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:07.800888 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:07.801356 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:07.801364 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:07.801367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:07.801638 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:07.801833 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:07.802582 (+ 749us) server_negotiation.cc:299] Negotiation successful
0929 16:42:07.802824 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":233,"threads_started":1}
I20260929 16:42:07.804270 7412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:07.783139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:33747 (local address 127.0.197.67:47083)
0929 16:42:07.784377 (+ 1238us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:07.784510 (+ 133us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:07.786409 (+ 1899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:07.787373 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:07.787384 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:07.788006 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:07.788658 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:07.788679 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:07.790328 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:07.790333 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:07.791840 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:07.791867 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:07.792885 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:07.792890 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:07.793121 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:07.794007 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:07.794036 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:07.797095 (+ 3059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:42:07.800096 (+ 3001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:07.800107 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:07.800136 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:07.800599 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:07.801009 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:07.801014 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:07.801018 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:07.801224 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:07.801772 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:42:07.801781 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:42:07.802193 (+ 412us) client_negotiation.cc:769] Sending connection context
0929 16:42:07.802478 (+ 285us) client_negotiation.cc:240] Negotiation successful
0929 16:42:07.803081 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":890,"thread_start_us":598,"threads_started":1}
I20260929 16:42:07.805732 7409 heartbeater.cc:347] Connected to a master server at 127.0.197.126:33747
I20260929 16:42:07.806058 7409 heartbeater.cc:464] Registering TS with master...
I20260929 16:42:07.806860 7409 heartbeater.cc:511] Master 127.0.197.126:33747 requested a full tablet report, sending...
I20260929 16:42:07.808599 6952 ts_manager.cc:194] Registered new tserver with Master: af179336d9f44d9ebe6d2205959274e4 (127.0.197.67:36295)
I20260929 16:42:07.809623 6952 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.197.67@KRBTEST.COM'} at 127.0.197.67:47083
I20260929 16:42:07.810190 789 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.197.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.197.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.197.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/oryx_127.0.197.125.keytab.
Entry for principal oryx/127.0.197.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/oryx_127.0.197.125.keytab.
I20260929 16:42:07.864305 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.125:34675
--webserver_interface=127.0.197.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/oryx_127.0.197.125.keytab
--principal=oryx/127.0.197.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:34055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.126:33747,127.0.197.125:34675
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:42:08.205421 7415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:42:08.205753 7415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:42:08.205911 7415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:42:08.213647 7415 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:42:08.213795 7415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:42:08.213843 7415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:42:08.213932 7415 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:42:08.214000 7415 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:42:08.224689 7415 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:34055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.197.126:33747,127.0.197.125:34675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/oryx_127.0.197.125.keytab
--principal=oryx/127.0.197.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.125:34675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.197.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
W20260929 16:42:08.225111 7415 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.197.126:33747,127.0.197.125:34675'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260929 16:42:08.226306 7415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:42:08.227806 7415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:42:08.240710 7420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:08.240628 7423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:08.240691 7421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:42:08.241706 7415 server_base.cc:1034] running on GCE node
I20260929 16:42:08.242630 7415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:42:08.244529 7415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:42:08.245890 7415 hybrid_clock.cc:648] HybridClock initialized: now 1790700128245810 us; error 77 us; skew 500 ppm
I20260929 16:42:08.246294 7415 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:42:08 dist-test-slave-l31f krb5kdc[6907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700128, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.197.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:42:08.249758 7415 init.cc:377] Logged in from keytab as oryx/127.0.197.125@KRBTEST.COM (short username oryx)
I20260929 16:42:08.251549 7415 webserver.cc:460] Webserver started at http://127.0.197.125:45987/ using document root <none> and password file <none>
I20260929 16:42:08.252287 7415 fs_manager.cc:359] Metadata directory not provided
I20260929 16:42:08.252386 7415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:42:08.252663 7415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:42:08.256443 7415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-1/data/instance:
uuid: "489538cf2b1e4ae68c3ae1c94128ff1f"
format_stamp: "Formatted at 2026-09-29 16:42:08 on dist-test-slave-l31f"
server_key: "7b7662e81548ca8dc17b528435c392a3"
server_key_iv: "ab5f40b164efbc3b8945e22fe9f1b72c"
server_key_version: "encryptionkey@0"
I20260929 16:42:08.257253 7415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-1/wal/instance:
uuid: "489538cf2b1e4ae68c3ae1c94128ff1f"
format_stamp: "Formatted at 2026-09-29 16:42:08 on dist-test-slave-l31f"
server_key: "7b7662e81548ca8dc17b528435c392a3"
server_key_iv: "ab5f40b164efbc3b8945e22fe9f1b72c"
server_key_version: "encryptionkey@0"
I20260929 16:42:08.259305 7275 heartbeater.cc:503] Master 127.0.197.126:33747 was elected leader, sending a full tablet report...
I20260929 16:42:08.263553 7415 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 16:42:08.267590 7430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:08.269312 7415 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:42:08.269490 7415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-1/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-1/wal
uuid: "489538cf2b1e4ae68c3ae1c94128ff1f"
format_stamp: "Formatted at 2026-09-29 16:42:08 on dist-test-slave-l31f"
server_key: "7b7662e81548ca8dc17b528435c392a3"
server_key_iv: "ab5f40b164efbc3b8945e22fe9f1b72c"
server_key_version: "encryptionkey@0"
I20260929 16:42:08.269693 7415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:42:08.291923 7415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:42:08.296546 7415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:42:08.296787 7415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:42:08.316756 7415 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.125:34675
I20260929 16:42:08.316772 7481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.125:34675 every 8 connection(s)
I20260929 16:42:08.318296 7415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-1/data/info.pb
I20260929 16:42:08.322432 7482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:42:08.327071 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 7415
I20260929 16:42:08.330127 7482 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33747 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:42:08 dist-test-slave-l31f krb5kdc[6907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.197.125@KRBTEST.COM
Sep 29 16:42:08 dist-test-slave-l31f krb5kdc[6907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700128, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.197.125@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM
I20260929 16:42:08.352663 7486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:08.334784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46092 (local address 127.0.197.126:33747)
0929 16:42:08.335377 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:08.335384 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:08.335978 (+ 594us) server_negotiation.cc:407] Connection header received
0929 16:42:08.336236 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:08.336240 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:08.336323 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:08.336421 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:08.337964 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:08.339230 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:08.341323 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:08.341932 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:08.342072 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:08.346314 (+ 4242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:08.346320 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:08.346323 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:08.346361 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:08.349368 (+ 3007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:08.350242 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:08.350249 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:08.350253 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:08.350395 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:08.350835 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:08.350842 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:08.350846 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:08.351267 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:08.351437 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:08.352017 (+ 580us) server_negotiation.cc:299] Negotiation successful
0929 16:42:08.352276 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":312,"threads_started":1}
I20260929 16:42:08.353952 7487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:08.334853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:33747 (local address 127.0.0.1:46092)
0929 16:42:08.335766 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:08.335833 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:08.336034 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:08.336542 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:08.336549 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:08.337433 (+ 884us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:08.337783 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:08.337796 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:08.339424 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:08.339437 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:08.341163 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:08.341176 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:08.342186 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:08.342191 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:08.342332 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:08.343337 (+ 1005us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:08.343384 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:08.346091 (+ 2707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:42:08.349576 (+ 3485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:08.349591 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:08.349606 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:08.350079 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:08.350523 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:08.350530 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:08.350534 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:08.350710 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:08.351421 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:42:08.351431 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:42:08.351640 (+ 209us) client_negotiation.cc:769] Sending connection context
0929 16:42:08.351932 (+ 292us) client_negotiation.cc:240] Negotiation successful
0929 16:42:08.352614 (+ 682us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"mutex_wait_us":140,"thread_start_us":468,"threads_started":1}
I20260929 16:42:08.353989 7485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:08.329643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57916 (local address 127.0.197.125:34675)
0929 16:42:08.330561 (+ 918us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:08.330578 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:08.330635 (+ 57us) server_negotiation.cc:407] Connection header received
0929 16:42:08.332493 (+ 1858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:08.332528 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:08.333102 (+ 574us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:08.333569 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:08.335533 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:08.337655 (+ 2122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:08.339647 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:08.340769 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:08.341008 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:08.346126 (+ 5118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:08.346137 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:08.346151 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:08.346187 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:08.349498 (+ 3311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:08.350255 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:08.350263 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:08.350265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:08.350418 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:08.350830 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:08.350837 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:08.350850 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:08.351457 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:08.351786 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:08.352100 (+ 314us) server_negotiation.cc:299] Negotiation successful
0929 16:42:08.352437 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":641,"thread_start_us":407,"threads_started":1}
I20260929 16:42:08.356178 7482 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.197.125" port: 34675 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:42:08 dist-test-slave-l31f krb5kdc[6907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700128, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.197.125@KRBTEST.COM for oryx/127.0.197.125@KRBTEST.COM
I20260929 16:42:08.369203 7487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:08.356902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.125:34675 (local address 127.0.0.1:57920)
0929 16:42:08.357152 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:08.357178 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:08.357291 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:08.357640 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:08.357643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:08.358016 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:08.358141 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:08.358149 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:08.359145 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:08.359151 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:08.360447 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:08.360459 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:08.361124 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:08.361129 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:08.361315 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:08.361929 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:08.361947 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:08.364038 (+ 2091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:42:08.366941 (+ 2903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:08.366946 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:08.366949 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:08.367435 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:08.367818 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:08.367822 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:08.367825 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:08.367953 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:08.368463 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:42:08.368467 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:42:08.368582 (+ 115us) client_negotiation.cc:769] Sending connection context
0929 16:42:08.368728 (+ 146us) client_negotiation.cc:240] Negotiation successful
0929 16:42:08.368960 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 16:42:08.369433 7485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:08.357082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57920 (local address 127.0.197.125:34675)
0929 16:42:08.357206 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:08.357210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:08.357247 (+ 37us) server_negotiation.cc:407] Connection header received
0929 16:42:08.357429 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:08.357434 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:08.357517 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:08.357625 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:08.358273 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:08.358979 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:08.360594 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:08.360984 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:08.361117 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:08.364248 (+ 3131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:08.364252 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:08.364263 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:08.364293 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:08.366795 (+ 2502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:08.367564 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:08.367568 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:08.367570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:08.367696 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:08.368053 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:08.368056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:08.368058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:08.368343 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:08.368483 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:08.368844 (+ 361us) server_negotiation.cc:299] Negotiation successful
0929 16:42:08.369098 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":30}
I20260929 16:42:08.374306 7482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f: Bootstrap starting.
I20260929 16:42:08.377657 7482 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f: Neither blocks nor log segments found. Creating new log.
I20260929 16:42:08.378796 7482 log.cc:824] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f: Log is configured to *not* fsync() on all Append() calls
I20260929 16:42:08.381839 7482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f: No bootstrap required, opened a new log
I20260929 16:42:08.389253 7482 raft_consensus.cc:348] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33747 } } peers { permanent_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 34675 } }
I20260929 16:42:08.389667 7482 raft_consensus.cc:374] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:42:08.389765 7482 raft_consensus.cc:729] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 489538cf2b1e4ae68c3ae1c94128ff1f, State: Initialized, Role: FOLLOWER
I20260929 16:42:08.390398 7482 consensus_queue.cc:260] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33747 } } peers { permanent_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 34675 } }
I20260929 16:42:08.391397 7489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33747 } } peers { permanent_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 34675 } } }
I20260929 16:42:08.391718 7489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:42:08.392493 7482 sys_catalog.cc:564] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f [sys.catalog]: configured and running, proceeding with master startup.
W20260929 16:42:08.402215 7500 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:42:08.402442 7500 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:42:08.405011 7415 master_runner.cc:428] Detected that this master 489538cf2b1e4ae68c3ae1c94128ff1f is joining an existing cluster
I20260929 16:42:08.405133 7415 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.197.125:34675
I20260929 16:42:08.427556 7486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:08.409967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46108 (local address 127.0.197.126:33747)
0929 16:42:08.410143 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:08.410148 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:08.410877 (+ 729us) server_negotiation.cc:407] Connection header received
0929 16:42:08.411203 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:08.411209 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:08.411329 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:08.411404 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:08.413251 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:08.414374 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:08.416610 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:08.417310 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:08.417433 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:08.420301 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:08.420312 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:08.420320 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:08.420378 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:08.424608 (+ 4230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:08.425406 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:08.425412 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:08.425416 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:08.425561 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:08.425962 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:08.425968 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:08.425971 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:08.426264 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:08.426425 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:08.426894 (+ 469us) server_negotiation.cc:299] Negotiation successful
0929 16:42:08.427159 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 16:42:08.427649 7506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:08.410031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:33747 (local address 127.0.0.1:46108)
0929 16:42:08.410752 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:08.410773 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:08.411028 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:08.411645 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:08.411649 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:08.412331 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:08.413008 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:08.413026 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:08.414591 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:08.414597 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:08.416394 (+ 1797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:08.416413 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:08.417762 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:08.417770 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:08.417952 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:08.418721 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:08.418748 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:08.420009 (+ 1261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:42:08.424773 (+ 4764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:08.424778 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:08.424782 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:08.425271 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:08.425682 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:08.425687 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:08.425691 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:08.425842 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:08.426387 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:42:08.426392 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:42:08.426585 (+ 193us) client_negotiation.cc:769] Sending connection context
0929 16:42:08.426773 (+ 188us) client_negotiation.cc:240] Negotiation successful
0929 16:42:08.427158 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":388,"threads_started":1}
I20260929 16:42:08.440416 7485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:08.419422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57932 (local address 127.0.197.125:34675)
0929 16:42:08.419725 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:08.419734 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:08.422698 (+ 2964us) server_negotiation.cc:407] Connection header received
0929 16:42:08.422905 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:08.422909 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:08.422974 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:08.423103 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:08.424147 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:08.425008 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:08.427134 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:08.427944 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:08.428047 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:08.431739 (+ 3692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:08.431745 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:08.431749 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:08.431796 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:08.435966 (+ 4170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:08.437289 (+ 1323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:08.437299 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:08.437305 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:08.437501 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:08.438017 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:08.438025 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:08.438031 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:08.438541 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:08.438745 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:08.439416 (+ 671us) server_negotiation.cc:299] Negotiation successful
0929 16:42:08.439779 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260929 16:42:08.441879 7507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:08.410997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.125:34675 (local address 127.0.0.1:57932)
0929 16:42:08.422572 (+ 11575us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:08.422598 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:08.422779 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:08.423238 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:08.423241 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:08.423736 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:08.423992 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:08.424002 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:08.425231 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:08.425237 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:08.426944 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:08.426959 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:08.428364 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:08.428371 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:08.428554 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:08.429422 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:08.429446 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:08.431523 (+ 2077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:42:08.436188 (+ 4665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:08.436196 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:08.436203 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:08.437120 (+ 917us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:08.437683 (+ 563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:08.437690 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:08.437696 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:08.437894 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:08.438833 (+ 939us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:42:08.438841 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:42:08.439065 (+ 224us) client_negotiation.cc:769] Sending connection context
0929 16:42:08.440727 (+ 1662us) client_negotiation.cc:240] Negotiation successful
0929 16:42:08.441128 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11364,"thread_start_us":11834,"threads_started":1}
Sep 29 16:42:08 dist-test-slave-l31f krb5kdc[6907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700125, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.197.126@KRBTEST.COM for oryx/127.0.197.125@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.197.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.197.124@KRBTEST.COM" created.
I20260929 16:42:08.464080 7485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:08.434854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57944 (local address 127.0.197.125:34675)
0929 16:42:08.440689 (+ 5835us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:08.440694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:08.440823 (+ 129us) server_negotiation.cc:407] Connection header received
0929 16:42:08.443414 (+ 2591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:08.443424 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:08.443626 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:08.443745 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:08.446381 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:08.448053 (+ 1672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:08.451533 (+ 3480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:08.452227 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:08.452368 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:08.457078 (+ 4710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:08.457083 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:08.457087 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:08.457124 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:08.460232 (+ 3108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:08.461309 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:08.461315 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:08.461319 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:08.461514 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:08.462069 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:08.462076 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:08.462082 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:08.462514 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:08.462688 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:08.463375 (+ 687us) server_negotiation.cc:299] Negotiation successful
0929 16:42:08.463663 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5728,"thread_start_us":16317,"threads_started":1}
I20260929 16:42:08.464195 7509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:08.434580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.125:34675 (local address 127.0.0.1:57944)
0929 16:42:08.440662 (+ 6082us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:08.440715 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:08.440982 (+ 267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:08.444217 (+ 3235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:08.444231 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:08.445564 (+ 1333us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:08.446065 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:08.446087 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:08.448338 (+ 2251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:08.448346 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:08.450677 (+ 2331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:08.450703 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:08.452742 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:08.452749 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:08.453002 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:08.453955 (+ 953us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:08.453976 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:08.456837 (+ 2861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:42:08.460435 (+ 3598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:08.460455 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:08.460463 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:08.461110 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:08.461672 (+ 562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:08.461680 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:08.461688 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:08.461905 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:08.462697 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:42:08.462709 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:42:08.462942 (+ 233us) client_negotiation.cc:769] Sending connection context
0929 16:42:08.463214 (+ 272us) client_negotiation.cc:240] Negotiation successful
0929 16:42:08.463665 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5734,"thread_start_us":12476,"threads_started":1}
I20260929 16:42:08.466722 6952 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.197.125:34675: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f"
member_type: NON_VOTER
last_known_addr {
host: "127.0.197.125"
port: 34675
}
attrs {
promote: true
}
}
dest_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f"
cas_config_opid_index: -1
I20260929 16:42:08.467990 6952 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33747 } } peers { permanent_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" member_type: NON_VOTER last_known_addr { host: "127.0.197.125" port: 34675 } attrs { promote: true } }
I20260929 16:42:08.479429 7457 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:42:08.481803 7513 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 489538cf2b1e4ae68c3ae1c94128ff1f (127.0.197.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33747 } } peers { permanent_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" member_type: NON_VOTER last_known_addr { host: "127.0.197.125" port: 34675 } attrs { promote: true } } }
I20260929 16:42:08.484331 7457 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f [term 1 FOLLOWER]: Refusing update from remote peer ab1f8d56bd194fb2b0c5bd8cc6c2993f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 16:42:08.486616 7515 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f [LEADER]: Connected to new peer: Peer: permanent_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" member_type: NON_VOTER last_known_addr { host: "127.0.197.125" port: 34675 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:42:08.488334 7513 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.197.125:34675
I20260929 16:42:08.489516 7415 master.cc:537] Master@127.0.197.125:34675 shutting down...
I20260929 16:42:08.490837 7514 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33747 } } peers { permanent_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" member_type: NON_VOTER last_known_addr { host: "127.0.197.125" port: 34675 } attrs { promote: true } } }
I20260929 16:42:08.491676 7514 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f [sys.catalog]: This master's current role is: LEADER
W20260929 16:42:08.492529 6937 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.197.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/oryx_127.0.197.124.keytab.
Entry for principal oryx/127.0.197.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/oryx_127.0.197.124.keytab.
I20260929 16:42:08.501878 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.124:36163
--webserver_interface=127.0.197.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/oryx_127.0.197.124.keytab
--principal=oryx/127.0.197.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:34055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.126:33747,127.0.197.125:34675,127.0.197.124:36163
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:42:08.512606 7415 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:42:08.513302 7415 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:42:08.513623 7415 tablet_replica.cc:330] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f: stopping tablet replica
W20260929 16:42:08.515951 6937 consensus_peers.cc:487] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f -> Peer 489538cf2b1e4ae68c3ae1c94128ff1f (127.0.197.125:34675): Couldn't send request to peer 489538cf2b1e4ae68c3ae1c94128ff1f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 16:42:08.521869 7415 master.cc:552] Master@127.0.197.125:34675 shutdown complete.
I20260929 16:42:08.521974 7415 master_runner.cc:305] Clearing existing system tablet
I20260929 16:42:08.522470 7415 fs_manager.cc:359] Metadata directory not provided
I20260929 16:42:08.522573 7415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:42:08.526078 7415 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.002s
I20260929 16:42:08.528033 7521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:08.528492 7415 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:42:08.528772 7415 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:42:08.528885 7415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-1/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-1/wal
uuid: "489538cf2b1e4ae68c3ae1c94128ff1f"
format_stamp: "Formatted at 2026-09-29 16:42:08 on dist-test-slave-l31f"
server_key: "7b7662e81548ca8dc17b528435c392a3"
server_key_iv: "ab5f40b164efbc3b8945e22fe9f1b72c"
server_key_version: "encryptionkey@0"
I20260929 16:42:08.534271 7415 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:42:08.541990 7415 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 16:42:08.542217 7415 log.cc:1196] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260929 16:42:08.542696 7415 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f: Deleting consensus metadata
I20260929 16:42:08.542982 7415 master_runner.cc:315] Copying system tablet from 127.0.197.126:33747
I20260929 16:42:08.552011 7415 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.197.126:33747
I20260929 16:42:08.567613 7486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:08.552878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46124 (local address 127.0.197.126:33747)
0929 16:42:08.553111 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:08.553120 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:08.555696 (+ 2576us) server_negotiation.cc:407] Connection header received
0929 16:42:08.555958 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:08.555966 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:08.556042 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:08.556167 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:08.557137 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:08.557881 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:08.559194 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:08.559634 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:08.559775 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:08.561883 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:08.561888 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:08.561892 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:08.561927 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:08.564726 (+ 2799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:08.565424 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:08.565431 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:08.565435 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:08.565573 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:08.565961 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:08.565968 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:08.565971 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:08.566267 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:08.566425 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:08.566987 (+ 562us) server_negotiation.cc:299] Negotiation successful
0929 16:42:08.567264 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 16:42:08.567600 7527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:08.552831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:33747 (local address 127.0.0.1:46124)
0929 16:42:08.555551 (+ 2720us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:08.555573 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:08.555781 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:08.556190 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:08.556194 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:08.556688 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:08.556971 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:08.556984 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:08.558050 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:08.558055 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:08.558960 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:08.558972 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:08.559810 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:08.559817 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:08.560005 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:08.560719 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:08.560737 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:08.561602 (+ 865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:42:08.564893 (+ 3291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:08.564899 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:08.564903 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:08.565287 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:08.565695 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:08.565700 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:08.565704 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:08.565848 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:08.566388 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:42:08.566393 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:42:08.566566 (+ 173us) client_negotiation.cc:769] Sending connection context
0929 16:42:08.566740 (+ 174us) client_negotiation.cc:240] Negotiation successful
0929 16:42:08.567131 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2563,"thread_start_us":2409,"threads_started":1}
I20260929 16:42:08.568817 6972 tablet_copy_service.cc:140] P ab1f8d56bd194fb2b0c5bd8cc6c2993f: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 489538cf2b1e4ae68c3ae1c94128ff1f ({username='oryx', principal='oryx/127.0.197.125@KRBTEST.COM'} at 127.0.0.1:46124)
I20260929 16:42:08.569100 6972 tablet_copy_service.cc:161] P ab1f8d56bd194fb2b0c5bd8cc6c2993f: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 489538cf2b1e4ae68c3ae1c94128ff1f at {username='oryx', principal='oryx/127.0.197.125@KRBTEST.COM'} at 127.0.0.1:46124: session id = 489538cf2b1e4ae68c3ae1c94128ff1f-00000000000000000000000000000000
I20260929 16:42:08.572274 6972 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 16:42:08.574374 7415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 16:42:08.579588 7415 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f: tablet copy: Starting download of 0 data blocks...
I20260929 16:42:08.579949 7415 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f: tablet copy: Starting download of 1 WAL segments...
I20260929 16:42:08.582726 7415 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 16:42:08.586381 6972 tablet_copy_service.cc:342] P ab1f8d56bd194fb2b0c5bd8cc6c2993f: Request end of tablet copy session 489538cf2b1e4ae68c3ae1c94128ff1f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.197.125@KRBTEST.COM'} at 127.0.0.1:46124
I20260929 16:42:08.586608 6972 tablet_copy_service.cc:433] P ab1f8d56bd194fb2b0c5bd8cc6c2993f: ending tablet copy session 489538cf2b1e4ae68c3ae1c94128ff1f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 489538cf2b1e4ae68c3ae1c94128ff1f
I20260929 16:42:08.597738 7415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:42:08.600734 7528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:08.605139 7529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:42:08.607410 7415 server_base.cc:1034] running on GCE node
W20260929 16:42:08.607812 7531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:42:08.608312 7415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:42:08.609007 7415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:42:08.610311 7415 hybrid_clock.cc:648] HybridClock initialized: now 1790700128610257 us; error 32 us; skew 500 ppm
I20260929 16:42:08.610555 7415 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:42:08 dist-test-slave-l31f krb5kdc[6907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700128, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.197.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:42:08.612581 7415 init.cc:377] Logged in from keytab as oryx/127.0.197.125@KRBTEST.COM (short username oryx)
I20260929 16:42:08.618911 7415 webserver.cc:460] Webserver started at http://127.0.197.125:32947/ using document root <none> and password file <none>
I20260929 16:42:08.619316 7415 fs_manager.cc:359] Metadata directory not provided
I20260929 16:42:08.619412 7415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:42:08.621898 7415 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 16:42:08.623728 7538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:08.624212 7415 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 16:42:08.624373 7415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-1/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-1/wal
uuid: "489538cf2b1e4ae68c3ae1c94128ff1f"
format_stamp: "Formatted at 2026-09-29 16:42:08 on dist-test-slave-l31f"
server_key: "7b7662e81548ca8dc17b528435c392a3"
server_key_iv: "ab5f40b164efbc3b8945e22fe9f1b72c"
server_key_version: "encryptionkey@0"
I20260929 16:42:08.624534 7415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:42:08.638037 7415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:42:08.642005 7415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:42:08.691447 7415 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.125:34675
I20260929 16:42:08.691459 7589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.125:34675 every 8 connection(s)
I20260929 16:42:08.692552 7415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-1/data/info.pb
I20260929 16:42:08.703975 7590 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:42:08.707146 7590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f: Bootstrap starting.
I20260929 16:42:08.721743 7590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:42:08.722276 7590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f: Bootstrap complete.
I20260929 16:42:08.723207 7590 raft_consensus.cc:348] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33747 } } peers { permanent_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" member_type: NON_VOTER last_known_addr { host: "127.0.197.125" port: 34675 } attrs { promote: true } }
I20260929 16:42:08.723479 7590 raft_consensus.cc:729] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 489538cf2b1e4ae68c3ae1c94128ff1f, State: Initialized, Role: LEARNER
I20260929 16:42:08.723788 7590 consensus_queue.cc:260] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33747 } } peers { permanent_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" member_type: NON_VOTER last_known_addr { host: "127.0.197.125" port: 34675 } attrs { promote: true } }
I20260929 16:42:08.724715 7593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33747 } } peers { permanent_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" member_type: NON_VOTER last_known_addr { host: "127.0.197.125" port: 34675 } attrs { promote: true } } }
I20260929 16:42:08.725021 7590 sys_catalog.cc:564] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:42:08.725281 7593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f [sys.catalog]: This master's current role is: LEARNER
I20260929 16:42:08.733191 7604 catalog_manager.cc:1259] Loaded cluster ID: f0d8d655a3754a4e8a727c9f26d8474b
I20260929 16:42:08.733448 7604 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f: loading cluster ID for follower catalog manager: success
I20260929 16:42:08.737035 7604 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f: acquiring CA information for follower catalog manager: success
I20260929 16:42:08.739315 7604 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 16:42:08.812750 7409 heartbeater.cc:503] Master 127.0.197.126:33747 was elected leader, sending a full tablet report...
W20260929 16:42:08.963346 7519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:42:08.963688 7519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:42:08.963779 7519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:42:08.971365 7519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:42:08.971493 7519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:42:08.971539 7519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:42:08.971596 7519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:42:08.971647 7519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:42:08.983371 7519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:34055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.197.126:33747,127.0.197.125:34675,127.0.197.124:36163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/oryx_127.0.197.124.keytab
--principal=oryx/127.0.197.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.124:36163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.197.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:42:08.984814 7519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:42:08.986238 7519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:42:09.001734 7609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:42:09.003355 7519 server_base.cc:1034] running on GCE node
W20260929 16:42:09.003660 7610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:09.004158 7612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:42:09.005829 7519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:42:09.007885 7519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:42:09.009177 7519 hybrid_clock.cc:648] HybridClock initialized: now 1790700129009116 us; error 59 us; skew 500 ppm
I20260929 16:42:09.009627 7519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:42:09 dist-test-slave-l31f krb5kdc[6907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700129, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.197.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:42:09.019565 7519 init.cc:377] Logged in from keytab as oryx/127.0.197.124@KRBTEST.COM (short username oryx)
I20260929 16:42:09.021332 7519 webserver.cc:460] Webserver started at http://127.0.197.124:35163/ using document root <none> and password file <none>
I20260929 16:42:09.022140 7519 fs_manager.cc:359] Metadata directory not provided
I20260929 16:42:09.022317 7519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:42:09.022647 7519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:42:09.026307 7519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-2/data/instance:
uuid: "f5404ea6fa1045c792ce56877881c332"
format_stamp: "Formatted at 2026-09-29 16:42:09 on dist-test-slave-l31f"
server_key: "bc261d8ace87a27dfbdfb6d048b167ad"
server_key_iv: "a0aca7345aef6e80e94bb7175a5b844a"
server_key_version: "encryptionkey@0"
I20260929 16:42:09.027375 7519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-2/wal/instance:
uuid: "f5404ea6fa1045c792ce56877881c332"
format_stamp: "Formatted at 2026-09-29 16:42:09 on dist-test-slave-l31f"
server_key: "bc261d8ace87a27dfbdfb6d048b167ad"
server_key_iv: "a0aca7345aef6e80e94bb7175a5b844a"
server_key_version: "encryptionkey@0"
I20260929 16:42:09.033210 7519 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:42:09.037668 7619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:09.039369 7519 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 16:42:09.039597 7519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-2/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-2/wal
uuid: "f5404ea6fa1045c792ce56877881c332"
format_stamp: "Formatted at 2026-09-29 16:42:09 on dist-test-slave-l31f"
server_key: "bc261d8ace87a27dfbdfb6d048b167ad"
server_key_iv: "a0aca7345aef6e80e94bb7175a5b844a"
server_key_version: "encryptionkey@0"
I20260929 16:42:09.039819 7519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:42:09.055562 7519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:42:09.060453 7519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:42:09.060768 7519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:42:09.082814 7519 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.124:36163
I20260929 16:42:09.082858 7670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.124:36163 every 8 connection(s)
I20260929 16:42:09.084415 7519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-2/data/info.pb
I20260929 16:42:09.088449 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 7519
I20260929 16:42:09.089226 7671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:42:09.103039 7671 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33747 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:42:09.106990 7676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:09.094327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.125:34675 (local address 127.0.0.1:57958)
0929 16:42:09.094820 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:09.094844 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:09.094998 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:09.100427 (+ 5429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:09.100434 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:09.100469 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:42:09.100690 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:09.100703 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:09.102621 (+ 1918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:09.102627 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:09.104622 (+ 1995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:09.104641 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:09.106098 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:09.106105 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:09.106306 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:09.106370 (+ 64us) client_negotiation.cc:769] Sending connection context
0929 16:42:09.106480 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 16:42:09.106629 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":619,"threads_started":1}
I20260929 16:42:09.107791 7677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:09.094720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57958 (local address 127.0.197.125:34675)
0929 16:42:09.100012 (+ 5292us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:09.100018 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:09.100038 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:42:09.100112 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:09.100120 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:09.100210 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:09.100345 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:42:09.100872 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:09.102438 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:09.104843 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:09.105863 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:09.106023 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:09.106270 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:09.106528 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 16:42:09.106694 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5141,"thread_start_us":5027,"threads_started":1}
I20260929 16:42:09.109148 7565 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260929 16:42:09.110189 7593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab1f8d56bd194fb2b0c5bd8cc6c2993f. Latest consensus state: current_term: 1 leader_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33747 } } peers { permanent_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" member_type: NON_VOTER last_known_addr { host: "127.0.197.125" port: 34675 } attrs { promote: true } } }
I20260929 16:42:09.110509 7593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f [sys.catalog]: This master's current role is: LEARNER
Sep 29 16:42:09 dist-test-slave-l31f krb5kdc[6907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.197.124@KRBTEST.COM
Sep 29 16:42:09 dist-test-slave-l31f krb5kdc[6907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700129, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.197.124@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM
I20260929 16:42:09.125007 7679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:09.106993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46136 (local address 127.0.197.126:33747)
0929 16:42:09.108279 (+ 1286us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:09.108285 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:09.108313 (+ 28us) server_negotiation.cc:407] Connection header received
0929 16:42:09.108353 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:09.108357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:09.108414 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:09.108540 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:09.110573 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:09.111878 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:09.113301 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:09.113726 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:09.113869 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:09.118556 (+ 4687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:09.118562 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:09.118566 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:09.118603 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:09.121550 (+ 2947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:09.122292 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:09.122300 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:09.122304 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:09.122876 (+ 572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:09.123436 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:09.123441 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:09.123444 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:09.123818 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:09.123971 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:09.124490 (+ 519us) server_negotiation.cc:299] Negotiation successful
0929 16:42:09.124663 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1117,"thread_start_us":1422,"threads_started":1}
I20260929 16:42:09.125492 7674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:09.090815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52170 (local address 127.0.197.124:36163)
0929 16:42:09.091781 (+ 966us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:09.091792 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:09.099690 (+ 7898us) server_negotiation.cc:407] Connection header received
0929 16:42:09.102492 (+ 2802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:09.102528 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:09.103080 (+ 552us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:09.103526 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:09.105343 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:09.107335 (+ 1992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:09.109655 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:09.111278 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:09.111508 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:09.116992 (+ 5484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:09.117005 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:09.117025 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:09.117072 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:09.120605 (+ 3533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:09.121405 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:09.121422 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:09.121430 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:09.121585 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:09.121986 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:09.121991 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:09.121994 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:09.122573 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:09.122931 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:09.123474 (+ 543us) server_negotiation.cc:299] Negotiation successful
0929 16:42:09.123975 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":741,"thread_start_us":492,"threads_started":1}
I20260929 16:42:09.125535 7678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:09.106429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:33747 (local address 127.0.0.1:46136)
0929 16:42:09.107950 (+ 1521us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:09.108005 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:09.108170 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:09.108822 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:09.108831 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:09.109799 (+ 968us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:09.110363 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:09.110381 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:09.112050 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:09.112055 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:09.113106 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:09.113118 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:09.113992 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:09.114002 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:09.114306 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:09.115372 (+ 1066us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:09.115399 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:09.118299 (+ 2900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:42:09.121701 (+ 3402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:09.121713 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:09.121720 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:09.122155 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:09.123053 (+ 898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:09.123060 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:09.123065 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:09.123266 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:09.123960 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:42:09.123968 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:42:09.124142 (+ 174us) client_negotiation.cc:769] Sending connection context
0929 16:42:09.124349 (+ 207us) client_negotiation.cc:240] Negotiation successful
0929 16:42:09.124688 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1240,"thread_start_us":1061,"threads_started":1}
I20260929 16:42:09.128161 7671 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.197.125" port: 34675 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:42:09 dist-test-slave-l31f krb5kdc[6907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700129, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.197.124@KRBTEST.COM for oryx/127.0.197.125@KRBTEST.COM
I20260929 16:42:09.144212 7677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:09.129148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57966 (local address 127.0.197.125:34675)
0929 16:42:09.129314 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:09.129319 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:09.129335 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:42:09.129385 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:09.129388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:09.129446 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:09.129527 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:09.130192 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:09.130937 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:09.132258 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:09.132675 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:09.132783 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:09.137243 (+ 4460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:09.137250 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:09.137254 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:09.137300 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:09.140682 (+ 3382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:09.141685 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:09.141690 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:09.141693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:09.141874 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:09.142353 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:09.142366 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:09.142368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:09.142671 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:09.142834 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:09.143657 (+ 823us) server_negotiation.cc:299] Negotiation successful
0929 16:42:09.143950 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 16:42:09.144340 7678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:09.128920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.125:34675 (local address 127.0.0.1:57966)
0929 16:42:09.129102 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:09.129119 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:09.129204 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:09.129562 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:09.129565 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:09.129910 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:09.130050 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:09.130058 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:09.131150 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:09.131154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:09.132095 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:09.132108 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:09.132907 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:09.132912 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:09.133080 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:09.133726 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:09.133748 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:09.136809 (+ 3061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:42:09.140931 (+ 4122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:09.140937 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:09.140940 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:09.141523 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:09.142000 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:09.142003 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:09.142007 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:09.142149 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:09.142971 (+ 822us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:42:09.142977 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:42:09.143320 (+ 343us) client_negotiation.cc:769] Sending connection context
0929 16:42:09.143525 (+ 205us) client_negotiation.cc:240] Negotiation successful
0929 16:42:09.143902 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 16:42:09.145562 7671 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.197.124" port: 36163 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:42:09 dist-test-slave-l31f krb5kdc[6907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700129, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.197.124@KRBTEST.COM for oryx/127.0.197.124@KRBTEST.COM
I20260929 16:42:09.160023 7678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:09.146521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:36163 (local address 127.0.0.1:52186)
0929 16:42:09.146722 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:09.146739 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:09.146860 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:09.147195 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:09.147199 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:09.147563 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:09.147743 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:09.147752 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:09.148878 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:09.148882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:09.149738 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:09.149749 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:09.150435 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:09.150439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:09.150591 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:09.151330 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:09.151347 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:09.153945 (+ 2598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:42:09.157215 (+ 3270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:09.157219 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:09.157222 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:09.157626 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:09.157996 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:09.157999 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:09.158001 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:09.158152 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:09.158816 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:42:09.158822 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:42:09.159278 (+ 456us) client_negotiation.cc:769] Sending connection context
0929 16:42:09.159408 (+ 130us) client_negotiation.cc:240] Negotiation successful
0929 16:42:09.159668 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 16:42:09.160054 7674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:09.146717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52186 (local address 127.0.197.124:36163)
0929 16:42:09.146876 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:09.146881 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:09.146904 (+ 23us) server_negotiation.cc:407] Connection header received
0929 16:42:09.146945 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:09.146948 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:09.147063 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:09.147150 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:09.147898 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:09.148751 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:09.149894 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:09.150297 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:09.150408 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:09.154233 (+ 3825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:09.154241 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:09.154246 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:09.154287 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:09.157055 (+ 2768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:09.157739 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:09.157749 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:09.157751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:09.157898 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:09.158272 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:09.158276 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:09.158278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:09.158575 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:09.158727 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:09.159518 (+ 791us) server_negotiation.cc:299] Negotiation successful
0929 16:42:09.159799 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":52}
I20260929 16:42:09.165226 7671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332: Bootstrap starting.
I20260929 16:42:09.168610 7671 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332: Neither blocks nor log segments found. Creating new log.
I20260929 16:42:09.169795 7671 log.cc:824] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332: Log is configured to *not* fsync() on all Append() calls
I20260929 16:42:09.172842 7671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332: No bootstrap required, opened a new log
I20260929 16:42:09.181056 7671 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33747 } } peers { permanent_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 34675 } } peers { permanent_uuid: "f5404ea6fa1045c792ce56877881c332" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 36163 } }
I20260929 16:42:09.181416 7671 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:42:09.181566 7671 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5404ea6fa1045c792ce56877881c332, State: Initialized, Role: FOLLOWER
I20260929 16:42:09.182294 7671 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33747 } } peers { permanent_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 34675 } } peers { permanent_uuid: "f5404ea6fa1045c792ce56877881c332" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 36163 } }
I20260929 16:42:09.183275 7681 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33747 } } peers { permanent_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 34675 } } peers { permanent_uuid: "f5404ea6fa1045c792ce56877881c332" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 36163 } } }
I20260929 16:42:09.183645 7681 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:42:09.184415 7671 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 16:42:09.194985 7692 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:42:09.195225 7692 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:42:09.198629 7681 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:42:09.198992 7681 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33747 } } peers { permanent_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 34675 } } peers { permanent_uuid: "f5404ea6fa1045c792ce56877881c332" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 36163 } }
I20260929 16:42:09.201203 7681 leader_election.cc:288] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab1f8d56bd194fb2b0c5bd8cc6c2993f (127.0.197.126:33747), 489538cf2b1e4ae68c3ae1c94128ff1f (127.0.197.125:34675)
I20260929 16:42:09.201614 7519 master_runner.cc:428] Detected that this master f5404ea6fa1045c792ce56877881c332 is joining an existing cluster
I20260929 16:42:09.201723 7519 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.197.124:36163
I20260929 16:42:09.201956 6962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f5404ea6fa1045c792ce56877881c332" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" is_pre_election: true
I20260929 16:42:09.202067 7565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f5404ea6fa1045c792ce56877881c332" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" is_pre_election: true
I20260929 16:42:09.203161 7621 leader_election.cc:302] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f5404ea6fa1045c792ce56877881c332; no voters: 489538cf2b1e4ae68c3ae1c94128ff1f, ab1f8d56bd194fb2b0c5bd8cc6c2993f
I20260929 16:42:09.203588 7681 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 16:42:09.231864 7679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:09.211329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46138 (local address 127.0.197.126:33747)
0929 16:42:09.211928 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:09.211937 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:09.212709 (+ 772us) server_negotiation.cc:407] Connection header received
0929 16:42:09.212965 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:09.212974 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:09.213086 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:09.213215 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:09.215333 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:09.216459 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:09.218845 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:09.219612 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:09.219726 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:09.224268 (+ 4542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:09.224276 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:09.224281 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:09.224324 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:09.228285 (+ 3961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:09.229230 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:09.229236 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:09.229240 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:09.229406 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:09.229912 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:09.229917 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:09.229921 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:09.230303 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:09.230550 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:09.231225 (+ 675us) server_negotiation.cc:299] Negotiation successful
0929 16:42:09.231478 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429}
I20260929 16:42:09.232918 7699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:09.210904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:33747 (local address 127.0.0.1:46138)
0929 16:42:09.212536 (+ 1632us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:09.212562 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:09.212798 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:09.213425 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:09.213429 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:09.214657 (+ 1228us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:09.215106 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:09.215128 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:09.216682 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:09.216689 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:09.218572 (+ 1883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:09.218592 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:09.220067 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:09.220074 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:09.220317 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:09.221130 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:09.221156 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:09.224018 (+ 2862us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:42:09.228491 (+ 4473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:09.228498 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:09.228502 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:09.229074 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:09.229557 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:09.229562 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:09.229565 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:09.229768 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:09.230647 (+ 879us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:42:09.230653 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:42:09.230877 (+ 224us) client_negotiation.cc:769] Sending connection context
0929 16:42:09.232055 (+ 1178us) client_negotiation.cc:240] Negotiation successful
0929 16:42:09.232402 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1431,"mutex_wait_us":1,"thread_start_us":2227,"threads_started":1}
I20260929 16:42:09.233035 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 7013
I20260929 16:42:09.234134 7674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:09.211679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52196 (local address 127.0.197.124:36163)
0929 16:42:09.211931 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:09.211941 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:09.214011 (+ 2070us) server_negotiation.cc:407] Connection header received
0929 16:42:09.214270 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:09.214279 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:09.214347 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:09.214428 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:09.216331 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:09.217910 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:09.221074 (+ 3164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:09.221864 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:09.221978 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:09.225210 (+ 3232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:09.225216 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:09.225222 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:09.225268 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:09.229414 (+ 4146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:09.230313 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:09.230318 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:09.230322 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:09.230712 (+ 390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:09.231807 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:09.231813 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:09.231816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:09.232600 (+ 784us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:09.232777 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:09.233570 (+ 793us) server_negotiation.cc:299] Negotiation successful
0929 16:42:09.233818 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 16:42:09.235430 7698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:09.210925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:36163 (local address 127.0.0.1:52196)
0929 16:42:09.213900 (+ 2975us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:09.213925 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:09.214106 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:09.214672 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:09.214680 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:09.215667 (+ 987us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:09.216120 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:09.216146 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:09.218212 (+ 2066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:09.218220 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:09.220913 (+ 2693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:09.220929 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:09.222763 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:09.222773 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:09.223033 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:09.223754 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:09.223778 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:09.224993 (+ 1215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:42:09.229561 (+ 4568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:09.229565 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:09.229569 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:09.230182 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:09.230878 (+ 696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:09.230883 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:09.230886 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:09.231675 (+ 789us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:09.232849 (+ 1174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:42:09.232854 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:42:09.233301 (+ 447us) client_negotiation.cc:769] Sending connection context
0929 16:42:09.234567 (+ 1266us) client_negotiation.cc:240] Negotiation successful
0929 16:42:09.234894 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2432,"mutex_wait_us":571,"spinlock_wait_cycles":10240,"thread_start_us":1082,"threads_started":1}
I20260929 16:42:09.248096 7677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:09.213766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57982 (local address 127.0.197.125:34675)
0929 16:42:09.214109 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:09.214115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:09.220835 (+ 6720us) server_negotiation.cc:407] Connection header received
0929 16:42:09.221099 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:09.221104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:09.221205 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:09.221285 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:09.223342 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:09.224487 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:09.226945 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:09.227659 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:09.227801 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:09.230445 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:09.230455 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:09.230464 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:09.230508 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:09.237933 (+ 7425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:09.244701 (+ 6768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:09.244711 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:09.244715 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:09.245111 (+ 396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:09.245979 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:09.245988 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:09.245992 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:09.246494 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:09.246656 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:09.247411 (+ 755us) server_negotiation.cc:299] Negotiation successful
0929 16:42:09.247695 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228}
I20260929 16:42:09.249604 7700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:09.214279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.125:34675 (local address 127.0.0.1:57982)
0929 16:42:09.220718 (+ 6439us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:09.220746 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:09.220941 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:09.221535 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:09.221540 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:09.222200 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:09.223145 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:09.223160 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:09.224727 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:09.224734 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:09.226747 (+ 2013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:09.226763 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:09.228151 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:09.228159 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:09.228361 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:09.228949 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:09.228977 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:09.230215 (+ 1238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:42:09.243902 (+ 13687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:09.243908 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:09.243912 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:09.244489 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:09.245343 (+ 854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:09.245349 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:09.245354 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:09.245759 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:09.246776 (+ 1017us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:42:09.246783 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:42:09.247070 (+ 287us) client_negotiation.cc:769] Sending connection context
0929 16:42:09.248752 (+ 1682us) client_negotiation.cc:240] Negotiation successful
0929 16:42:09.249120 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6291,"thread_start_us":6134,"threads_started":1}
Sep 29 16:42:09 dist-test-slave-l31f krb5kdc[6907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700125, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.197.126@KRBTEST.COM for oryx/127.0.197.124@KRBTEST.COM
I20260929 16:42:09.260202 7676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:09.240084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:36163 (local address 127.0.0.1:52200)
0929 16:42:09.240458 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:09.240477 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:09.240624 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:09.241448 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:09.241456 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:09.242154 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:09.242466 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:09.242483 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:09.244724 (+ 2241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:09.244735 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:09.246696 (+ 1961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:09.246722 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:09.248372 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:09.248381 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:09.248739 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:09.249481 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:09.249506 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:09.252674 (+ 3168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:42:09.257198 (+ 4524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:09.257204 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:09.257208 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:09.257623 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:09.258023 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:09.258028 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:09.258032 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:09.258199 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:09.258762 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:42:09.258767 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:42:09.259298 (+ 531us) client_negotiation.cc:769] Sending connection context
0929 16:42:09.259471 (+ 173us) client_negotiation.cc:240] Negotiation successful
0929 16:42:09.259821 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192}
I20260929 16:42:09.260200 7674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:09.240190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52200 (local address 127.0.197.124:36163)
0929 16:42:09.240947 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:09.240956 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:09.240979 (+ 23us) server_negotiation.cc:407] Connection header received
0929 16:42:09.241103 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:09.241111 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:09.241230 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:09.242714 (+ 1484us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:09.242978 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:09.244470 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:09.247060 (+ 2590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:09.248132 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:09.252708 (+ 4576us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:09.252951 (+ 243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:09.252957 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:09.252962 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:09.252995 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:09.257031 (+ 4036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:09.257757 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:09.257764 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:09.257767 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:09.257896 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:09.258312 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:09.258319 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:09.258322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:09.258620 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:09.258779 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:09.259570 (+ 791us) server_negotiation.cc:299] Negotiation successful
0929 16:42:09.259826 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594}
Sep 29 16:42:09 dist-test-slave-l31f krb5kdc[6907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.197.65@KRBTEST.COM
I20260929 16:42:09.261758 6952 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.197.124:36163: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f5404ea6fa1045c792ce56877881c332"
member_type: NON_VOTER
last_known_addr {
host: "127.0.197.124"
port: 36163
}
attrs {
promote: true
}
}
dest_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f"
cas_config_opid_index: 5
I20260929 16:42:09.262684 6952 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33747 } } peers { permanent_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" member_type: NON_VOTER last_known_addr { host: "127.0.197.125" port: 34675 } attrs { promote: true } } peers { permanent_uuid: "f5404ea6fa1045c792ce56877881c332" member_type: NON_VOTER last_known_addr { host: "127.0.197.124" port: 36163 } attrs { promote: true } }
I20260929 16:42:09.265659 7675 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f5404ea6fa1045c792ce56877881c332 (127.0.197.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33747 } } peers { permanent_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" member_type: NON_VOTER last_known_addr { host: "127.0.197.125" port: 34675 } attrs { promote: true } } peers { permanent_uuid: "f5404ea6fa1045c792ce56877881c332" member_type: NON_VOTER last_known_addr { host: "127.0.197.124" port: 36163 } attrs { promote: true } } }
I20260929 16:42:09.272238 7565 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f [term 1 LEARNER]: Refusing update from remote peer ab1f8d56bd194fb2b0c5bd8cc6c2993f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 16:42:09.273144 7701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:09.234433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37878 (local address 127.0.197.65:35927)
0929 16:42:09.248788 (+ 14355us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:09.248797 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:09.248835 (+ 38us) server_negotiation.cc:407] Connection header received
0929 16:42:09.248914 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:09.248924 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:09.249269 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:09.249463 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:09.250969 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:09.252505 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:09.254657 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:09.255754 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:09.255913 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:09.262990 (+ 7077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:09.263028 (+ 38us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:09.263049 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:09.263098 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:09.267905 (+ 4807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:09.268986 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:09.268999 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:09.269003 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:09.269204 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:09.269724 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:09.269730 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:09.269733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:09.270207 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:09.272077 (+ 1870us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:09.272281 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 16:42:09.272686 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14139,"thread_start_us":15727,"threads_started":1}
I20260929 16:42:09.275486 7136 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:42:09.275938 7675 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.197.124:36163
I20260929 16:42:09.276055 7646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:42:09.276650 7519 master.cc:537] Master@127.0.197.124:36163 shutting down...
I20260929 16:42:09.280479 7646 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332 [term 1 FOLLOWER]: Refusing update from remote peer ab1f8d56bd194fb2b0c5bd8cc6c2993f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 16:42:09.285449 7675 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33747 } } peers { permanent_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" member_type: NON_VOTER last_known_addr { host: "127.0.197.125" port: 34675 } attrs { promote: true } } peers { permanent_uuid: "f5404ea6fa1045c792ce56877881c332" member_type: NON_VOTER last_known_addr { host: "127.0.197.124" port: 36163 } attrs { promote: true } } }
I20260929 16:42:09.285929 7675 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f [sys.catalog]: This master's current role is: LEADER
I20260929 16:42:09.289598 7702 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f [LEADER]: Connected to new peer: Peer: permanent_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" member_type: NON_VOTER last_known_addr { host: "127.0.197.125" port: 34675 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 16:42:09.296272 7705 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5404ea6fa1045c792ce56877881c332" member_type: NON_VOTER last_known_addr { host: "127.0.197.124" port: 36163 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:42:09.301149 7565 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f5404ea6fa1045c792ce56877881c332 (127.0.197.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33747 } } peers { permanent_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" member_type: NON_VOTER last_known_addr { host: "127.0.197.125" port: 34675 } attrs { promote: true } } peers { permanent_uuid: "f5404ea6fa1045c792ce56877881c332" member_type: NON_VOTER last_known_addr { host: "127.0.197.124" port: 36163 } attrs { promote: true } } }
I20260929 16:42:09.307474 7593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33747 } } peers { permanent_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" member_type: NON_VOTER last_known_addr { host: "127.0.197.125" port: 34675 } attrs { promote: true } } peers { permanent_uuid: "f5404ea6fa1045c792ce56877881c332" member_type: NON_VOTER last_known_addr { host: "127.0.197.124" port: 36163 } attrs { promote: true } } }
I20260929 16:42:09.307874 7593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f [sys.catalog]: This master's current role is: LEARNER
I20260929 16:42:09.309135 7705 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f: attempting to promote NON_VOTER 489538cf2b1e4ae68c3ae1c94128ff1f to VOTER
I20260929 16:42:09.310191 7705 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33747 } } peers { permanent_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "f5404ea6fa1045c792ce56877881c332" member_type: NON_VOTER last_known_addr { host: "127.0.197.124" port: 36163 } attrs { promote: true } }
I20260929 16:42:09.313238 7565 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f [term 1 LEARNER]: Refusing update from remote peer ab1f8d56bd194fb2b0c5bd8cc6c2993f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260929 16:42:09.313920 7705 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f [LEADER]: Connected to new peer: Peer: permanent_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 34675 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 16:42:09.317408 7702 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 489538cf2b1e4ae68c3ae1c94128ff1f (127.0.197.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33747 } } peers { permanent_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "f5404ea6fa1045c792ce56877881c332" member_type: NON_VOTER last_known_addr { host: "127.0.197.124" port: 36163 } attrs { promote: true } } }
I20260929 16:42:09.319396 7565 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 489538cf2b1e4ae68c3ae1c94128ff1f (127.0.197.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33747 } } peers { permanent_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "f5404ea6fa1045c792ce56877881c332" member_type: NON_VOTER last_known_addr { host: "127.0.197.124" port: 36163 } attrs { promote: true } } }
W20260929 16:42:09.322631 7625 tablet.cc:2369] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:42:09.324088 7593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33747 } } peers { permanent_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "f5404ea6fa1045c792ce56877881c332" member_type: NON_VOTER last_known_addr { host: "127.0.197.124" port: 36163 } attrs { promote: true } } }
I20260929 16:42:09.324412 7593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:42:09.326670 7703 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33747 } } peers { permanent_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "f5404ea6fa1045c792ce56877881c332" member_type: NON_VOTER last_known_addr { host: "127.0.197.124" port: 36163 } attrs { promote: true } } }
I20260929 16:42:09.327139 7703 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f [sys.catalog]: This master's current role is: LEADER
I20260929 16:42:09.327584 7703 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33747 } } peers { permanent_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "f5404ea6fa1045c792ce56877881c332" member_type: NON_VOTER last_known_addr { host: "127.0.197.124" port: 36163 } attrs { promote: true } } }
I20260929 16:42:09.327797 7703 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f [sys.catalog]: This master's current role is: LEADER
W20260929 16:42:09.328599 6938 consensus_peers.cc:487] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f -> Peer f5404ea6fa1045c792ce56877881c332 (127.0.197.124:36163): Couldn't send request to peer f5404ea6fa1045c792ce56877881c332. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 16:42:09.332567 7519 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:42:09.333007 7519 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:42:09.333236 7519 tablet_replica.cc:330] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332: stopping tablet replica
I20260929 16:42:09.339555 7519 master.cc:552] Master@127.0.197.124:36163 shutdown complete.
I20260929 16:42:09.339686 7519 master_runner.cc:305] Clearing existing system tablet
I20260929 16:42:09.340067 7519 fs_manager.cc:359] Metadata directory not provided
I20260929 16:42:09.340196 7519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:42:09.342581 7519 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260929 16:42:09.344202 7710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:09.344559 7519 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:42:09.344864 7519 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 16:42:09.344961 7519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-2/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-2/wal
uuid: "f5404ea6fa1045c792ce56877881c332"
format_stamp: "Formatted at 2026-09-29 16:42:09 on dist-test-slave-l31f"
server_key: "bc261d8ace87a27dfbdfb6d048b167ad"
server_key_iv: "a0aca7345aef6e80e94bb7175a5b844a"
server_key_version: "encryptionkey@0"
I20260929 16:42:09.348860 7519 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:42:09.354488 7519 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 16:42:09.354707 7519 log.cc:1196] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260929 16:42:09.355306 7519 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332: Deleting consensus metadata
I20260929 16:42:09.355677 7519 master_runner.cc:315] Copying system tablet from 127.0.197.126:33747
I20260929 16:42:09.358412 7519 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332: tablet copy: Beginning tablet copy session from remote peer at address 127.0.197.126:33747
I20260929 16:42:09.371487 7679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:09.359310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46150 (local address 127.0.197.126:33747)
0929 16:42:09.359565 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:09.359575 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:09.360075 (+ 500us) server_negotiation.cc:407] Connection header received
0929 16:42:09.360306 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:09.360311 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:09.360389 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:09.360501 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:09.361344 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:09.362088 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:09.363500 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:09.363962 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:09.364117 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:09.365968 (+ 1851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:09.365974 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:09.365979 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:09.366022 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:09.368706 (+ 2684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:09.369385 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:09.369392 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:09.369401 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:09.369528 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:09.369930 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:09.369936 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:09.369940 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:09.370245 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:09.370402 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:09.370826 (+ 424us) server_negotiation.cc:299] Negotiation successful
0929 16:42:09.371072 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260929 16:42:09.371518 7716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:09.359387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:33747 (local address 127.0.0.1:46150)
0929 16:42:09.359959 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:09.359979 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:09.360155 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:09.360524 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:09.360528 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:09.360991 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:09.361194 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:09.361202 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:09.362261 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:09.362267 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:09.363349 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:09.363362 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:09.364142 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:09.364148 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:09.364345 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:09.365006 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:09.365023 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:09.365769 (+ 746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:42:09.368870 (+ 3101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:09.368875 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:09.368880 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:09.369256 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:09.369640 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:09.369645 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:09.369649 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:09.369820 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:09.370365 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:42:09.370370 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:42:09.370547 (+ 177us) client_negotiation.cc:769] Sending connection context
0929 16:42:09.370711 (+ 164us) client_negotiation.cc:240] Negotiation successful
0929 16:42:09.371087 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":339,"threads_started":1}
I20260929 16:42:09.372421 6972 tablet_copy_service.cc:140] P ab1f8d56bd194fb2b0c5bd8cc6c2993f: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f5404ea6fa1045c792ce56877881c332 ({username='oryx', principal='oryx/127.0.197.124@KRBTEST.COM'} at 127.0.0.1:46150)
I20260929 16:42:09.372599 6972 tablet_copy_service.cc:161] P ab1f8d56bd194fb2b0c5bd8cc6c2993f: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f5404ea6fa1045c792ce56877881c332 at {username='oryx', principal='oryx/127.0.197.124@KRBTEST.COM'} at 127.0.0.1:46150: session id = f5404ea6fa1045c792ce56877881c332-00000000000000000000000000000000
I20260929 16:42:09.375139 6972 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 16:42:09.377244 7519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 16:42:09.382901 7519 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332: tablet copy: Starting download of 0 data blocks...
I20260929 16:42:09.383255 7519 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332: tablet copy: Starting download of 1 WAL segments...
I20260929 16:42:09.385823 7519 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 16:42:09.389746 6972 tablet_copy_service.cc:342] P ab1f8d56bd194fb2b0c5bd8cc6c2993f: Request end of tablet copy session f5404ea6fa1045c792ce56877881c332-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.197.124@KRBTEST.COM'} at 127.0.0.1:46150
I20260929 16:42:09.389909 6972 tablet_copy_service.cc:433] P ab1f8d56bd194fb2b0c5bd8cc6c2993f: ending tablet copy session f5404ea6fa1045c792ce56877881c332-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f5404ea6fa1045c792ce56877881c332
I20260929 16:42:09.401904 7519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:42:09.405833 7717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:09.408499 7718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:09.419431 7720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:42:09.419773 7519 server_base.cc:1034] running on GCE node
I20260929 16:42:09.420351 7519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:42:09.421087 7519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:42:09.422317 7519 hybrid_clock.cc:648] HybridClock initialized: now 1790700129422274 us; error 44 us; skew 500 ppm
I20260929 16:42:09.422583 7519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:42:09 dist-test-slave-l31f krb5kdc[6907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700129, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.197.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:42:09.424948 7519 init.cc:377] Logged in from keytab as oryx/127.0.197.124@KRBTEST.COM (short username oryx)
I20260929 16:42:09.426358 7519 webserver.cc:460] Webserver started at http://127.0.197.124:43621/ using document root <none> and password file <none>
I20260929 16:42:09.426714 7519 fs_manager.cc:359] Metadata directory not provided
I20260929 16:42:09.426817 7519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:42:09.430243 7519 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260929 16:42:09.432312 7727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:09.432797 7519 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 16:42:09.432976 7519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-2/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-2/wal
uuid: "f5404ea6fa1045c792ce56877881c332"
format_stamp: "Formatted at 2026-09-29 16:42:09 on dist-test-slave-l31f"
server_key: "bc261d8ace87a27dfbdfb6d048b167ad"
server_key_iv: "a0aca7345aef6e80e94bb7175a5b844a"
server_key_version: "encryptionkey@0"
I20260929 16:42:09.433243 7519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:42:09.448586 7519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:42:09.452644 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 7013
I20260929 16:42:09.453646 7519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:42:09.474339 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.65:35927
--local_ip_for_outbound_sockets=127.0.197.65
--tserver_master_addrs=127.0.197.126:33747
--webserver_port=40677
--webserver_interface=127.0.197.65
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/oryx_127.0.197.65.keytab
--principal=oryx/127.0.197.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:34055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.197.126:33747,127.0.197.125:34675
--tserver_master_addrs=127.0.197.126:33747,127.0.197.125:34675,127.0.197.124:36163 with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:42:09.483965 7519 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.124:36163
I20260929 16:42:09.483989 7779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.124:36163 every 8 connection(s)
I20260929 16:42:09.485045 7519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/master-2/data/info.pb
I20260929 16:42:09.490845 7780 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:42:09.495644 7780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332: Bootstrap starting.
I20260929 16:42:09.518086 7780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:42:09.518687 7780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332: Bootstrap complete.
I20260929 16:42:09.519797 7780 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33747 } } peers { permanent_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "f5404ea6fa1045c792ce56877881c332" member_type: NON_VOTER last_known_addr { host: "127.0.197.124" port: 36163 } attrs { promote: true } }
I20260929 16:42:09.520099 7780 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f5404ea6fa1045c792ce56877881c332, State: Initialized, Role: LEARNER
I20260929 16:42:09.520358 7780 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33747 } } peers { permanent_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "f5404ea6fa1045c792ce56877881c332" member_type: NON_VOTER last_known_addr { host: "127.0.197.124" port: 36163 } attrs { promote: true } }
I20260929 16:42:09.521191 7783 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33747 } } peers { permanent_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "f5404ea6fa1045c792ce56877881c332" member_type: NON_VOTER last_known_addr { host: "127.0.197.124" port: 36163 } attrs { promote: true } } }
I20260929 16:42:09.521631 7783 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332 [sys.catalog]: This master's current role is: LEARNER
I20260929 16:42:09.521476 7780 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:42:09.535836 7794 catalog_manager.cc:1259] Loaded cluster ID: f0d8d655a3754a4e8a727c9f26d8474b
I20260929 16:42:09.535936 7794 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332: loading cluster ID for follower catalog manager: success
I20260929 16:42:09.539983 7794 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332: acquiring CA information for follower catalog manager: success
I20260929 16:42:09.542068 7794 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 16:42:09.823987 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:09.817759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52202 (local address 127.0.197.124:36163)
0929 16:42:09.818414 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:09.818428 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:09.818443 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:42:09.818500 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:09.818503 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:09.818570 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:09.818704 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:42:09.819306 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:09.820304 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:09.821948 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:09.822829 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:09.822980 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:09.823253 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:09.823463 (+ 210us) server_negotiation.cc:299] Negotiation successful
0929 16:42:09.823620 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":357,"threads_started":1}
I20260929 16:42:09.823993 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:09.816344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:36163 (local address 127.0.0.1:52202)
0929 16:42:09.818098 (+ 1754us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:09.818123 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:09.818262 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:09.818723 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:09.818728 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:09.818753 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:42:09.818970 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:09.819121 (+ 151us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:09.820477 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:09.820486 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:09.821782 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:09.821796 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:09.823040 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:09.823048 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:09.823263 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:09.823309 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 16:42:09.823437 (+ 128us) client_negotiation.cc:240] Negotiation successful
0929 16:42:09.823617 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1629,"thread_start_us":1537,"threads_started":1}
I20260929 16:42:09.825730 7754 raft_consensus.cc:1203] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260929 16:42:09.826753 7783 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab1f8d56bd194fb2b0c5bd8cc6c2993f. Latest consensus state: current_term: 1 leader_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33747 } } peers { permanent_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "f5404ea6fa1045c792ce56877881c332" member_type: NON_VOTER last_known_addr { host: "127.0.197.124" port: 36163 } attrs { promote: true } } }
I20260929 16:42:09.827186 7783 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332 [sys.catalog]: This master's current role is: LEARNER
W20260929 16:42:09.884111 7773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:42:09.884477 7773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:42:09.884650 7773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:42:09.892679 7773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:42:09.892891 7773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:42:09.893072 7773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.65
I20260929 16:42:09.903553 7773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:34055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/oryx_127.0.197.65.keytab
--principal=oryx/127.0.197.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.65:35927
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.197.65
--webserver_port=40677
--tserver_master_addrs=127.0.197.126:33747,127.0.197.125:34675,127.0.197.124:36163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.65
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:42:09.905045 7773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:42:09.906405 7773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:42:09.921119 7804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:09.921196 7801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:09.921196 7802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:42:09.922261 7773 server_base.cc:1034] running on GCE node
I20260929 16:42:09.922986 7773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:42:09.924118 7773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:42:09.925340 7773 hybrid_clock.cc:648] HybridClock initialized: now 1790700129925296 us; error 36 us; skew 500 ppm
I20260929 16:42:09.925729 7773 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:42:09 dist-test-slave-l31f krb5kdc[6907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700129, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.197.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:42:09.929293 7773 init.cc:377] Logged in from keytab as oryx/127.0.197.65@KRBTEST.COM (short username oryx)
I20260929 16:42:09.930841 7773 webserver.cc:460] Webserver started at http://127.0.197.65:40677/ using document root <none> and password file <none>
I20260929 16:42:09.931612 7773 fs_manager.cc:359] Metadata directory not provided
I20260929 16:42:09.931803 7773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:42:09.938357 7773 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 16:42:09.941623 7811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:09.943171 7773 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 16:42:09.943329 7773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-0/wal
uuid: "52819fca8649475f98ef4c458a2e9a43"
format_stamp: "Formatted at 2026-09-29 16:42:06 on dist-test-slave-l31f"
server_key: "9ccc0bf0d0c14f8ae66feb3cdc130f28"
server_key_iv: "c3b9bf2ee5f60f4b4f1abe75f5d6390a"
server_key_version: "encryptionkey@0"
I20260929 16:42:09.944334 7773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:42:09.962462 7773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:42:09.967213 7773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:42:09.967546 7773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:42:09.969105 7773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:42:09.970654 7773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:42:09.970754 7773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:09.970882 7773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:42:09.970965 7773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:10.011556 7773 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.65:35927
I20260929 16:42:10.011601 7923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.65:35927 every 8 connection(s)
I20260929 16:42:10.013163 7773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
I20260929 16:42:10.015502 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 7773
I20260929 16:42:10.015950 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 7147
Sep 29 16:42:10 dist-test-slave-l31f krb5kdc[6907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.197.66@KRBTEST.COM
Sep 29 16:42:10 dist-test-slave-l31f krb5kdc[6907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700129, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.197.65@KRBTEST.COM for oryx/127.0.197.125@KRBTEST.COM
Sep 29 16:42:10 dist-test-slave-l31f krb5kdc[6907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700129, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.197.65@KRBTEST.COM for oryx/127.0.197.124@KRBTEST.COM
Sep 29 16:42:10 dist-test-slave-l31f krb5kdc[6907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700129, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.197.65@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM
I20260929 16:42:10.058004 7930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:10.017222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38868 (local address 127.0.197.66:44381)
0929 16:42:10.031622 (+ 14400us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:10.031630 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:10.031663 (+ 33us) server_negotiation.cc:407] Connection header received
0929 16:42:10.031730 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:10.031737 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:10.031965 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:10.032177 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:10.034801 (+ 2624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:10.036481 (+ 1680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:10.038498 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:10.039861 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:10.040668 (+ 807us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:10.047159 (+ 6491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:10.047181 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:10.047201 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:10.047256 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:10.051961 (+ 4705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:10.054949 (+ 2988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:10.054959 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:10.054963 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:10.055180 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:10.055731 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:10.055738 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:10.055742 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:10.056242 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:10.056553 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:10.057233 (+ 680us) server_negotiation.cc:299] Negotiation successful
0929 16:42:10.057550 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14203,"thread_start_us":14022,"threads_started":1}
I20260929 16:42:10.060976 7270 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:42:10.062785 7929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:10.017058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:58403 (local address 127.0.197.125:34675)
0929 16:42:10.036921 (+ 19863us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:10.036927 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:10.037780 (+ 853us) server_negotiation.cc:407] Connection header received
0929 16:42:10.037849 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:10.037854 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:10.037958 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:10.038062 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:10.043321 (+ 5259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:10.044545 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:10.048214 (+ 3669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:10.049059 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:10.049156 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:10.054758 (+ 5602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:10.054765 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:10.054770 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:10.054820 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:10.058564 (+ 3744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:10.059686 (+ 1122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:10.059693 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:10.059698 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:10.059852 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:10.060438 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:10.060442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:10.060445 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:10.060820 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:10.060961 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:10.062095 (+ 1134us) server_negotiation.cc:299] Negotiation successful
0929 16:42:10.062362 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19722,"thread_start_us":19607,"threads_started":1}
I20260929 16:42:10.063387 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:10.017414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:51141 (local address 127.0.197.124:36163)
0929 16:42:10.017611 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:10.017620 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:10.033537 (+ 15917us) server_negotiation.cc:407] Connection header received
0929 16:42:10.039279 (+ 5742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:10.039285 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:10.039382 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:10.039464 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:10.043327 (+ 3863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:10.044528 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:10.048214 (+ 3686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:10.048905 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:10.049024 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:10.055124 (+ 6100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:10.055133 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:10.055137 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:10.055178 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:10.060308 (+ 5130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:10.061101 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:10.061106 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:10.061109 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:10.061260 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:10.061773 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:10.061780 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:10.061783 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:10.062113 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:10.062232 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:10.062776 (+ 544us) server_negotiation.cc:299] Negotiation successful
0929 16:42:10.063067 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 16:42:10.065510 7932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:10.017699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:36163 (local address 127.0.197.65:51141)
0929 16:42:10.033289 (+ 15590us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:10.033360 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:10.036207 (+ 2847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:10.039675 (+ 3468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:10.039680 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:10.041732 (+ 2052us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:10.043066 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:10.043085 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:10.044812 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:10.044819 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:10.048018 (+ 3199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:10.048033 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:10.049498 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:10.049507 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:10.050241 (+ 734us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:10.051345 (+ 1104us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:10.051374 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:10.054824 (+ 3450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:42:10.060460 (+ 5636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:10.060464 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:10.060467 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:10.060967 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:10.061410 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:10.061415 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:10.061418 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:10.061618 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:10.062314 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:42:10.062318 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:42:10.062484 (+ 166us) client_negotiation.cc:769] Sending connection context
0929 16:42:10.063641 (+ 1157us) client_negotiation.cc:240] Negotiation successful
0929 16:42:10.064176 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15191,"mutex_wait_us":931,"spinlock_wait_cycles":12672,"thread_start_us":22089,"threads_started":1}
I20260929 16:42:10.065533 7934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:10.034968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:33747 (local address 127.0.197.65:60455)
0929 16:42:10.039633 (+ 4665us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:10.039662 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:10.039810 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:10.041066 (+ 1256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:10.041070 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:10.041825 (+ 755us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:10.043031 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:10.043068 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:10.044772 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:10.044811 (+ 39us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:10.048003 (+ 3192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:10.048027 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:10.049700 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:10.049711 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:10.049965 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:10.050889 (+ 924us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:10.050913 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:10.056254 (+ 5341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:42:10.061097 (+ 4843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:10.061105 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:10.061111 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:10.061811 (+ 700us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:10.062412 (+ 601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:10.062420 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:10.062426 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:10.062664 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:10.063569 (+ 905us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:42:10.063576 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:42:10.063830 (+ 254us) client_negotiation.cc:769] Sending connection context
0929 16:42:10.064032 (+ 202us) client_negotiation.cc:240] Negotiation successful
0929 16:42:10.064424 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3724,"mutex_wait_us":625,"thread_start_us":4234,"threads_started":1}
I20260929 16:42:10.066548 7933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:10.034992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:60455 (local address 127.0.197.126:33747)
0929 16:42:10.040739 (+ 5747us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:10.040745 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:10.040769 (+ 24us) server_negotiation.cc:407] Connection header received
0929 16:42:10.040853 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:10.040856 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:10.040915 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:10.041754 (+ 839us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:10.043293 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:10.044499 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:10.048213 (+ 3714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:10.049306 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:10.049440 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:10.056514 (+ 7074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:10.056521 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:10.056525 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:10.056571 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:10.060914 (+ 4343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:10.062033 (+ 1119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:10.062042 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:10.062048 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:10.062249 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:10.062822 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:10.062830 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:10.062836 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:10.063318 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:10.063493 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:10.065623 (+ 2130us) server_negotiation.cc:299] Negotiation successful
0929 16:42:10.065965 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5632,"thread_start_us":5470,"threads_started":1}
I20260929 16:42:10.067409 7924 heartbeater.cc:347] Connected to a master server at 127.0.197.124:36163
I20260929 16:42:10.067818 7924 heartbeater.cc:464] Registering TS with master...
I20260929 16:42:10.068962 7924 heartbeater.cc:511] Master 127.0.197.124:36163 requested a full tablet report, sending...
I20260929 16:42:10.070226 7925 heartbeater.cc:347] Connected to a master server at 127.0.197.126:33747
I20260929 16:42:10.070381 7925 heartbeater.cc:464] Registering TS with master...
I20260929 16:42:10.070968 7925 heartbeater.cc:511] Master 127.0.197.126:33747 requested a full tablet report, sending...
I20260929 16:42:10.071067 7931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:10.017853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.125:34675 (local address 127.0.197.65:58403)
0929 16:42:10.037373 (+ 19520us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:10.037395 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:10.037534 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:10.038644 (+ 1110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:10.038658 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:10.042364 (+ 3706us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:10.043031 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:10.043068 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:10.044775 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:10.044780 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:10.048017 (+ 3237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:10.048033 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:10.049498 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:10.049548 (+ 50us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:10.049953 (+ 405us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:10.050779 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:10.050826 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:10.054410 (+ 3584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:42:10.058742 (+ 4332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:10.058754 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:10.058778 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:10.059517 (+ 739us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:10.060022 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:10.060027 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:10.060031 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:10.060298 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:10.061063 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:42:10.061074 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:42:10.061622 (+ 548us) client_negotiation.cc:769] Sending connection context
0929 16:42:10.063300 (+ 1678us) client_negotiation.cc:240] Negotiation successful
0929 16:42:10.067374 (+ 4074us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19386,"mutex_wait_us":3352,"spinlock_wait_cycles":76032,"thread_start_us":14938,"threads_started":1}
I20260929 16:42:10.071857 7744 ts_manager.cc:194] Registered new tserver with Master: 52819fca8649475f98ef4c458a2e9a43 (127.0.197.65:35927)
I20260929 16:42:10.072326 6952 ts_manager.cc:194] Re-registered known tserver with Master: 52819fca8649475f98ef4c458a2e9a43 (127.0.197.65:35927)
I20260929 16:42:10.072520 7926 heartbeater.cc:347] Connected to a master server at 127.0.197.125:34675
I20260929 16:42:10.072649 7926 heartbeater.cc:464] Registering TS with master...
I20260929 16:42:10.073150 7926 heartbeater.cc:511] Master 127.0.197.125:34675 requested a full tablet report, sending...
I20260929 16:42:10.073251 6952 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.197.65@KRBTEST.COM'} at 127.0.197.65:60455
I20260929 16:42:10.074828 7555 ts_manager.cc:194] Registered new tserver with Master: 52819fca8649475f98ef4c458a2e9a43 (127.0.197.65:35927)
I20260929 16:42:10.158730 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 7147
I20260929 16:42:10.175743 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.66:44381
--local_ip_for_outbound_sockets=127.0.197.66
--tserver_master_addrs=127.0.197.126:33747
--webserver_port=44309
--webserver_interface=127.0.197.66
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/oryx_127.0.197.66.keytab
--principal=oryx/127.0.197.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:34055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.197.126:33747,127.0.197.125:34675
--tserver_master_addrs=127.0.197.126:33747,127.0.197.125:34675,127.0.197.124:36163 with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:42:10.293151 7702 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33747 } } peers { permanent_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "f5404ea6fa1045c792ce56877881c332" member_type: NON_VOTER last_known_addr { host: "127.0.197.124" port: 36163 } attrs { promote: true } } }
I20260929 16:42:10.293632 7702 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f [sys.catalog]: This master's current role is: LEADER
I20260929 16:42:10.295502 7937 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f: attempting to promote NON_VOTER f5404ea6fa1045c792ce56877881c332 to VOTER
I20260929 16:42:10.296108 7937 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33747 } } peers { permanent_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "f5404ea6fa1045c792ce56877881c332" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 36163 } attrs { promote: false } }
I20260929 16:42:10.298081 7565 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f [term 1 FOLLOWER]: Refusing update from remote peer ab1f8d56bd194fb2b0c5bd8cc6c2993f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260929 16:42:10.298071 7754 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332 [term 1 LEARNER]: Refusing update from remote peer ab1f8d56bd194fb2b0c5bd8cc6c2993f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260929 16:42:10.298764 7937 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f [LEADER]: Connected to new peer: Peer: permanent_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 34675 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 16:42:10.299265 7938 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5404ea6fa1045c792ce56877881c332" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 36163 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 16:42:10.302429 7937 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f5404ea6fa1045c792ce56877881c332 (127.0.197.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33747 } } peers { permanent_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "f5404ea6fa1045c792ce56877881c332" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 36163 } attrs { promote: false } } }
I20260929 16:42:10.303438 7565 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f5404ea6fa1045c792ce56877881c332 (127.0.197.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33747 } } peers { permanent_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "f5404ea6fa1045c792ce56877881c332" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 36163 } attrs { promote: false } } }
I20260929 16:42:10.308282 7754 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f5404ea6fa1045c792ce56877881c332 (127.0.197.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33747 } } peers { permanent_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "f5404ea6fa1045c792ce56877881c332" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 36163 } attrs { promote: false } } }
I20260929 16:42:10.308737 7941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33747 } } peers { permanent_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "f5404ea6fa1045c792ce56877881c332" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 36163 } attrs { promote: false } } }
I20260929 16:42:10.309108 7941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:42:10.312191 7783 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33747 } } peers { permanent_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "f5404ea6fa1045c792ce56877881c332" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 36163 } attrs { promote: false } } }
I20260929 16:42:10.312077 7702 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33747 } } peers { permanent_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "f5404ea6fa1045c792ce56877881c332" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 36163 } attrs { promote: false } } }
I20260929 16:42:10.312460 7783 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:42:10.312493 7702 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f [sys.catalog]: This master's current role is: LEADER
I20260929 16:42:10.312955 7702 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33747 } } peers { permanent_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "f5404ea6fa1045c792ce56877881c332" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 36163 } attrs { promote: false } } }
I20260929 16:42:10.312906 7937 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33747 } } peers { permanent_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "f5404ea6fa1045c792ce56877881c332" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 36163 } attrs { promote: false } } }
I20260929 16:42:10.313215 7702 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f [sys.catalog]: This master's current role is: LEADER
I20260929 16:42:10.313246 7937 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f [sys.catalog]: This master's current role is: LEADER
W20260929 16:42:10.538002 7936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:42:10.538319 7936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:42:10.538484 7936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:42:10.548194 7936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:42:10.548327 7936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:42:10.548555 7936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.66
I20260929 16:42:10.559265 7936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:34055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/oryx_127.0.197.66.keytab
--principal=oryx/127.0.197.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.66:44381
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.197.66
--webserver_port=44309
--tserver_master_addrs=127.0.197.126:33747,127.0.197.125:34675,127.0.197.124:36163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.66
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:42:10.560714 7936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:42:10.562110 7936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:42:10.576380 7950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:10.576442 7948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:10.576465 7947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:42:10.577282 7936 server_base.cc:1034] running on GCE node
I20260929 16:42:10.577981 7936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:42:10.579134 7936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:42:10.580390 7936 hybrid_clock.cc:648] HybridClock initialized: now 1790700130580337 us; error 51 us; skew 500 ppm
I20260929 16:42:10.580740 7936 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:42:10 dist-test-slave-l31f krb5kdc[6907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700130, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.197.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:42:10.584044 7936 init.cc:377] Logged in from keytab as oryx/127.0.197.66@KRBTEST.COM (short username oryx)
I20260929 16:42:10.585691 7936 webserver.cc:460] Webserver started at http://127.0.197.66:44309/ using document root <none> and password file <none>
I20260929 16:42:10.586414 7936 fs_manager.cc:359] Metadata directory not provided
I20260929 16:42:10.586532 7936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:42:10.592525 7936 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:42:10.595943 7957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:10.597700 7936 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:42:10.597887 7936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-1/wal
uuid: "9bed1bd04d834cdd9d75106656df7153"
format_stamp: "Formatted at 2026-09-29 16:42:07 on dist-test-slave-l31f"
server_key: "6c5b41f72cba70e07cc0aad0e22a1405"
server_key_iv: "dbaf7929c03853ab73005d00c0b6cc0c"
server_key_version: "encryptionkey@0"
I20260929 16:42:10.598935 7936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:42:10.643450 7936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:42:10.648658 7936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:42:10.648957 7936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:42:10.650050 7936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:42:10.651871 7936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:42:10.652009 7936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:10.652154 7936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:42:10.652251 7936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:10.687659 7936 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.66:44381
I20260929 16:42:10.687762 8069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.66:44381 every 8 connection(s)
I20260929 16:42:10.689254 7936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
I20260929 16:42:10.689709 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 7936
I20260929 16:42:10.690001 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 7281
Sep 29 16:42:10 dist-test-slave-l31f krb5kdc[6907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.197.67@KRBTEST.COM
I20260929 16:42:10.733075 8073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:10.691174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33254 (local address 127.0.197.67:36295)
0929 16:42:10.692457 (+ 1283us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:10.692466 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:10.703347 (+ 10881us) server_negotiation.cc:407] Connection header received
0929 16:42:10.704395 (+ 1048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:10.704406 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:10.704694 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:10.704832 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:10.707915 (+ 3083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:10.709556 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:10.711567 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:10.712560 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:10.712679 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:10.722373 (+ 9694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:10.722402 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:10.722430 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:10.722493 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:10.728039 (+ 5546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:10.729777 (+ 1738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:10.729787 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:10.729791 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:10.729972 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:10.730867 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:10.730873 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:10.730876 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:10.731385 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:10.731679 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:10.732356 (+ 677us) server_negotiation.cc:299] Negotiation successful
0929 16:42:10.732654 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1039,"thread_start_us":1449,"threads_started":1}
I20260929 16:42:10.735510 7404 generic_service.cc:196] Checking for leaks (request via RPC)
Sep 29 16:42:10 dist-test-slave-l31f krb5kdc[6907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700130, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.197.66@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM
Sep 29 16:42:10 dist-test-slave-l31f krb5kdc[6907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700130, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.197.66@KRBTEST.COM for oryx/127.0.197.124@KRBTEST.COM
Sep 29 16:42:10 dist-test-slave-l31f krb5kdc[6907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700130, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.197.66@KRBTEST.COM for oryx/127.0.197.125@KRBTEST.COM
I20260929 16:42:10.750500 8077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:10.708326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:49235 (local address 127.0.197.126:33747)
0929 16:42:10.712692 (+ 4366us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:10.712709 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:10.715559 (+ 2850us) server_negotiation.cc:407] Connection header received
0929 16:42:10.715641 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:10.715649 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:10.715765 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:10.715890 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:10.722901 (+ 7011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:10.724254 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:10.731254 (+ 7000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:10.731892 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:10.732020 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:10.739956 (+ 7936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:10.739967 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:10.739972 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:10.740027 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:10.744031 (+ 4004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:10.745853 (+ 1822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:10.745859 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:10.745862 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:10.746148 (+ 286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:10.746797 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:10.746806 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:10.746813 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:10.747406 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:10.748252 (+ 846us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:10.749710 (+ 1458us) server_negotiation.cc:299] Negotiation successful
0929 16:42:10.749972 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4231,"thread_start_us":4124,"threads_started":1}
I20260929 16:42:10.751703 8074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:10.692825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:52013 (local address 127.0.197.124:36163)
0929 16:42:10.703798 (+ 10973us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:10.703804 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:10.711075 (+ 7271us) server_negotiation.cc:407] Connection header received
0929 16:42:10.716155 (+ 5080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:10.716163 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:10.716263 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:10.716360 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:10.724009 (+ 7649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:10.725555 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:10.731545 (+ 5990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:10.732421 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:10.732536 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:10.739796 (+ 7260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:10.739805 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:10.739812 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:10.739861 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:10.743775 (+ 3914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:10.745845 (+ 2070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:10.745851 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:10.745855 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:10.746021 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:10.747030 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:10.747036 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:10.747039 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:10.747431 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:10.747581 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:10.750944 (+ 3363us) server_negotiation.cc:299] Negotiation successful
0929 16:42:10.751215 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10849,"thread_start_us":10737,"threads_started":1}
I20260929 16:42:10.753679 8075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:10.693652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:36163 (local address 127.0.197.66:52013)
0929 16:42:10.710393 (+ 16741us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:10.710461 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:10.714210 (+ 3749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:10.716611 (+ 2401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:10.716617 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:10.717828 (+ 1211us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:10.719110 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:10.719149 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:10.726379 (+ 7230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:10.726386 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:10.731385 (+ 4999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:10.731395 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:10.732674 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:10.732684 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:10.732941 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:10.734682 (+ 1741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:10.734715 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:10.739458 (+ 4743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:42:10.744030 (+ 4572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:10.744041 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:10.744063 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:10.745116 (+ 1053us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:10.746179 (+ 1063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:10.746183 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:10.746186 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:10.746870 (+ 684us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:10.747701 (+ 831us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:42:10.747713 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:42:10.748649 (+ 936us) client_negotiation.cc:769] Sending connection context
0929 16:42:10.752007 (+ 3358us) client_negotiation.cc:240] Negotiation successful
0929 16:42:10.752359 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16327,"mutex_wait_us":1,"spinlock_wait_cycles":4864,"thread_start_us":20003,"threads_started":1}
I20260929 16:42:10.754283 8078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:10.708324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:33747 (local address 127.0.197.66:49235)
0929 16:42:10.715108 (+ 6784us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:10.715138 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:10.715298 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:10.716347 (+ 1049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:10.716377 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:10.719996 (+ 3619us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:10.722692 (+ 2696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:10.722714 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:10.725830 (+ 3116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:10.725840 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:10.731083 (+ 5243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:10.731102 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:10.733405 (+ 2303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:10.733411 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:10.733917 (+ 506us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:10.734592 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:10.734632 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:10.739715 (+ 5083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:42:10.744297 (+ 4582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:10.744303 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:10.744306 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:10.745614 (+ 1308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:10.746277 (+ 663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:10.746304 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:10.746307 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:10.746603 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:10.749017 (+ 2414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:42:10.749025 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:42:10.749184 (+ 159us) client_negotiation.cc:769] Sending connection context
0929 16:42:10.749355 (+ 171us) client_negotiation.cc:240] Negotiation successful
0929 16:42:10.749968 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6585,"mutex_wait_us":23,"spinlock_wait_cycles":1047296,"thread_start_us":6819,"threads_started":1}
I20260929 16:42:10.755640 8070 heartbeater.cc:347] Connected to a master server at 127.0.197.124:36163
I20260929 16:42:10.755751 8071 heartbeater.cc:347] Connected to a master server at 127.0.197.126:33747
I20260929 16:42:10.756026 8070 heartbeater.cc:464] Registering TS with master...
I20260929 16:42:10.756031 8071 heartbeater.cc:464] Registering TS with master...
I20260929 16:42:10.757184 8080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:10.717914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:59047 (local address 127.0.197.125:34675)
0929 16:42:10.729928 (+ 12014us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:10.729934 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:10.732712 (+ 2778us) server_negotiation.cc:407] Connection header received
0929 16:42:10.732979 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:10.732986 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:10.733073 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:10.733188 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:10.734671 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:10.736312 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:10.737947 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:10.738534 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:10.738662 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:10.746586 (+ 7924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:10.746593 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:10.746598 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:10.746655 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:10.752441 (+ 5786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:10.753439 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:10.753445 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:10.753448 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:10.753603 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:10.755153 (+ 1550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:10.755164 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:10.755167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:10.755561 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:10.755707 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:10.756539 (+ 832us) server_negotiation.cc:299] Negotiation successful
0929 16:42:10.756796 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11853,"thread_start_us":11994,"threads_started":1}
I20260929 16:42:10.757159 8071 heartbeater.cc:511] Master 127.0.197.126:33747 requested a full tablet report, sending...
I20260929 16:42:10.757182 8070 heartbeater.cc:511] Master 127.0.197.124:36163 requested a full tablet report, sending...
I20260929 16:42:10.758733 8081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:10.720211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.125:34675 (local address 127.0.197.66:59047)
0929 16:42:10.732610 (+ 12399us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:10.732632 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:10.732826 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:10.733432 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:10.733436 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:10.734084 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:10.734415 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:10.734430 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:10.736513 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:10.736520 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:10.737772 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:10.737793 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:10.738982 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:10.738988 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:10.739248 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:10.740029 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:10.740050 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:10.746314 (+ 6264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:42:10.752713 (+ 6399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:10.752719 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:10.752727 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:10.753284 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:10.754105 (+ 821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:10.754111 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:10.754115 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:10.754857 (+ 742us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:10.755910 (+ 1053us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:42:10.755918 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:42:10.756202 (+ 284us) client_negotiation.cc:769] Sending connection context
0929 16:42:10.757680 (+ 1478us) client_negotiation.cc:240] Negotiation successful
0929 16:42:10.758003 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12243,"spinlock_wait_cycles":66816,"thread_start_us":12021,"threads_started":1}
I20260929 16:42:10.759074 6952 ts_manager.cc:194] Re-registered known tserver with Master: 9bed1bd04d834cdd9d75106656df7153 (127.0.197.66:44381)
I20260929 16:42:10.759668 7744 ts_manager.cc:194] Registered new tserver with Master: 9bed1bd04d834cdd9d75106656df7153 (127.0.197.66:44381)
I20260929 16:42:10.760041 6952 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.197.66@KRBTEST.COM'} at 127.0.197.66:49235
I20260929 16:42:10.761649 8076 heartbeater.cc:347] Connected to a master server at 127.0.197.125:34675
I20260929 16:42:10.761833 8076 heartbeater.cc:464] Registering TS with master...
I20260929 16:42:10.762337 8076 heartbeater.cc:511] Master 127.0.197.125:34675 requested a full tablet report, sending...
I20260929 16:42:10.763792 7555 ts_manager.cc:194] Registered new tserver with Master: 9bed1bd04d834cdd9d75106656df7153 (127.0.197.66:44381)
I20260929 16:42:10.839291 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 7281
W20260929 16:42:10.856120 6939 connection.cc:535] server connection from 127.0.197.67:47083 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 16:42:10.856889 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.67:36295
--local_ip_for_outbound_sockets=127.0.197.67
--tserver_master_addrs=127.0.197.126:33747
--webserver_port=42887
--webserver_interface=127.0.197.67
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/oryx_127.0.197.67.keytab
--principal=oryx/127.0.197.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:34055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.197.126:33747,127.0.197.125:34675
--tserver_master_addrs=127.0.197.126:33747,127.0.197.125:34675,127.0.197.124:36163 with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:42:11.075865 7925 heartbeater.cc:503] Master 127.0.197.126:33747 was elected leader, sending a full tablet report...
W20260929 16:42:11.221623 8083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:42:11.222036 8083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:42:11.222220 8083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:42:11.230581 8083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:42:11.230787 8083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:42:11.230984 8083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.67
I20260929 16:42:11.241712 8083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:34055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/krb5kdc/oryx_127.0.197.67.keytab
--principal=oryx/127.0.197.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.67:36295
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.197.67
--webserver_port=42887
--tserver_master_addrs=127.0.197.126:33747,127.0.197.125:34675,127.0.197.124:36163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.67
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:42:11.243269 8083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:42:11.244756 8083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:42:11.260435 8088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:11.260478 8089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:11.261111 8091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:42:11.261965 8083 server_base.cc:1034] running on GCE node
I20260929 16:42:11.262687 8083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:42:11.263932 8083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:42:11.265239 8083 hybrid_clock.cc:648] HybridClock initialized: now 1790700131265194 us; error 41 us; skew 500 ppm
I20260929 16:42:11.265641 8083 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:42:11 dist-test-slave-l31f krb5kdc[6907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700131, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.197.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:42:11.269269 8083 init.cc:377] Logged in from keytab as oryx/127.0.197.67@KRBTEST.COM (short username oryx)
I20260929 16:42:11.271198 8083 webserver.cc:460] Webserver started at http://127.0.197.67:42887/ using document root <none> and password file <none>
I20260929 16:42:11.271993 8083 fs_manager.cc:359] Metadata directory not provided
I20260929 16:42:11.272167 8083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:42:11.278295 8083 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 16:42:11.281976 8098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:11.283720 8083 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:42:11.283962 8083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-2/wal
uuid: "af179336d9f44d9ebe6d2205959274e4"
format_stamp: "Formatted at 2026-09-29 16:42:07 on dist-test-slave-l31f"
server_key: "102cba67c703b9db4b69cd2e9e617615"
server_key_iv: "0411ff8c1c934eb05d0b12bec7bca09f"
server_key_version: "encryptionkey@0"
I20260929 16:42:11.285085 8083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:42:11.305672 8083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:42:11.310514 8083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:42:11.310840 8083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:42:11.312101 8083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:42:11.313643 8083 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:42:11.313786 8083 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:11.313938 8083 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:42:11.314031 8083 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:11.352865 8083 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.67:36295
I20260929 16:42:11.352922 8210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.67:36295 every 8 connection(s)
I20260929 16:42:11.354653 8083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
I20260929 16:42:11.355924 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 8083
Sep 29 16:42:11 dist-test-slave-l31f krb5kdc[6907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700131, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.197.67@KRBTEST.COM for oryx/127.0.197.124@KRBTEST.COM
Sep 29 16:42:11 dist-test-slave-l31f krb5kdc[6907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700131, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.197.67@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM
I20260929 16:42:11.403231 8219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:11.359276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:33747 (local address 127.0.197.67:43991)
0929 16:42:11.372023 (+ 12747us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:11.372043 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:11.372270 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:11.372940 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:11.372947 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:11.373763 (+ 816us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:11.374165 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:11.374179 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:11.376084 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:11.376093 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:11.380327 (+ 4234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:11.380351 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:11.382310 (+ 1959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:11.382321 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:11.383655 (+ 1334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:11.386087 (+ 2432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:11.386121 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:11.389925 (+ 3804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:42:11.396692 (+ 6767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:11.396700 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:11.396706 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:11.397308 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:11.398062 (+ 754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:11.398070 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:11.398075 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:11.398359 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:11.399564 (+ 1205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:42:11.399577 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:42:11.400183 (+ 606us) client_negotiation.cc:769] Sending connection context
0929 16:42:11.400582 (+ 399us) client_negotiation.cc:240] Negotiation successful
0929 16:42:11.401331 (+ 749us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12611,"mutex_wait_us":1792,"thread_start_us":12007,"threads_started":1}
I20260929 16:42:11.403952 8217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:11.358141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:43991 (local address 127.0.197.126:33747)
0929 16:42:11.370389 (+ 12248us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:11.370398 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:11.372150 (+ 1752us) server_negotiation.cc:407] Connection header received
0929 16:42:11.372424 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:11.372429 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:11.372529 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:11.372632 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:11.374372 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:11.375806 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:11.380529 (+ 4723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:11.381275 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:11.381444 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:11.390233 (+ 8789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:11.390244 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:11.390252 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:11.390311 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:11.395918 (+ 5607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:11.397472 (+ 1554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:11.397482 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:11.397487 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:11.397685 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:11.398498 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:11.398507 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:11.398512 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:11.399022 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:11.399207 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:11.403158 (+ 3951us) server_negotiation.cc:299] Negotiation successful
0929 16:42:11.403512 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12065,"thread_start_us":11967,"threads_started":1}
I20260929 16:42:11.405877 8216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:11.358054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:44441 (local address 127.0.197.124:36163)
0929 16:42:11.368963 (+ 10909us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:11.368973 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:11.369005 (+ 32us) server_negotiation.cc:407] Connection header received
0929 16:42:11.369090 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:11.369098 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:11.369208 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:11.369379 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:11.374041 (+ 4662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:11.375799 (+ 1758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:11.380608 (+ 4809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:11.381660 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:11.381827 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:11.390269 (+ 8442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:11.390280 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:11.390288 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:11.390347 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:11.395365 (+ 5018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:11.397502 (+ 2137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:11.397510 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:11.397516 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:11.397913 (+ 397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:11.398814 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:11.398819 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:11.398822 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:11.399288 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:11.399459 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:11.404979 (+ 5520us) server_negotiation.cc:299] Negotiation successful
0929 16:42:11.405362 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10712,"thread_start_us":11469,"threads_started":1}
I20260929 16:42:11.407722 8218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:11.358933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:36163 (local address 127.0.197.67:44441)
0929 16:42:11.362047 (+ 3114us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:11.362129 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:11.366133 (+ 4004us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:11.371320 (+ 5187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:11.371330 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:11.372329 (+ 999us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:11.373692 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:11.373732 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:11.376115 (+ 2383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:11.376124 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:11.379773 (+ 3649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:11.379825 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:11.382336 (+ 2511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:11.382346 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:11.382855 (+ 509us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:11.384115 (+ 1260us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:11.384168 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:11.388809 (+ 4641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:42:11.395652 (+ 6843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:11.395667 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:11.395695 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:11.396538 (+ 843us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:11.398509 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:11.398515 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:11.398518 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:11.398685 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:11.403977 (+ 5292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:42:11.403983 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:42:11.404535 (+ 552us) client_negotiation.cc:769] Sending connection context
0929 16:42:11.406547 (+ 2012us) client_negotiation.cc:240] Negotiation successful
0929 16:42:11.407062 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2607,"mutex_wait_us":9,"thread_start_us":1950,"threads_started":1}
I20260929 16:42:11.409662 8212 heartbeater.cc:347] Connected to a master server at 127.0.197.126:33747
I20260929 16:42:11.409710 8211 heartbeater.cc:347] Connected to a master server at 127.0.197.124:36163
I20260929 16:42:11.410048 8212 heartbeater.cc:464] Registering TS with master...
I20260929 16:42:11.410053 8211 heartbeater.cc:464] Registering TS with master...
I20260929 16:42:11.411964 8212 heartbeater.cc:511] Master 127.0.197.126:33747 requested a full tablet report, sending...
I20260929 16:42:11.413146 8211 heartbeater.cc:511] Master 127.0.197.124:36163 requested a full tablet report, sending...
I20260929 16:42:11.413902 6952 ts_manager.cc:194] Re-registered known tserver with Master: af179336d9f44d9ebe6d2205959274e4 (127.0.197.67:36295)
I20260929 16:42:11.414443 7744 ts_manager.cc:194] Registered new tserver with Master: af179336d9f44d9ebe6d2205959274e4 (127.0.197.67:36295)
I20260929 16:42:11.415060 6952 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.197.67@KRBTEST.COM'} at 127.0.197.67:43991
Sep 29 16:42:11 dist-test-slave-l31f krb5kdc[6907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700131, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.197.67@KRBTEST.COM for oryx/127.0.197.125@KRBTEST.COM
I20260929 16:42:11.426966 8223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:11.372408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.125:34675 (local address 127.0.197.67:36517)
0929 16:42:11.377458 (+ 5050us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:11.377482 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:11.377631 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:11.408190 (+ 30559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:11.408198 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:11.408849 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:11.409183 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:11.409199 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:11.410815 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:11.410822 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:11.412711 (+ 1889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:11.412728 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:11.414612 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:11.414621 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:11.414802 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:11.415565 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:11.415593 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:11.418651 (+ 3058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:42:11.423524 (+ 4873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:11.423530 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:11.423534 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:11.424080 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:11.424562 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:11.424567 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:11.424570 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:11.424774 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:11.425570 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:42:11.425577 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:42:11.425773 (+ 196us) client_negotiation.cc:769] Sending connection context
0929 16:42:11.426055 (+ 282us) client_negotiation.cc:240] Negotiation successful
0929 16:42:11.426459 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4896,"thread_start_us":5331,"threads_started":1}
I20260929 16:42:11.428995 8228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:11.372412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:36517 (local address 127.0.197.125:34675)
0929 16:42:11.406642 (+ 34230us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:11.406649 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:11.406671 (+ 22us) server_negotiation.cc:407] Connection header received
0929 16:42:11.406865 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:11.406889 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:11.407157 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:11.409354 (+ 2197us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:11.409520 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:11.410592 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:11.413205 (+ 2613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:11.414322 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:11.414483 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:11.418920 (+ 4437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:11.418929 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:11.418934 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:11.418980 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:11.423348 (+ 4368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:11.424248 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:11.424254 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:11.424258 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:11.424409 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:11.424920 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:11.424926 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:11.424929 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:11.425351 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:11.425501 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:11.427316 (+ 1815us) server_negotiation.cc:299] Negotiation successful
0929 16:42:11.427555 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34084,"thread_start_us":39059,"threads_started":1}
I20260929 16:42:11.431705 8213 heartbeater.cc:347] Connected to a master server at 127.0.197.125:34675
I20260929 16:42:11.431886 8213 heartbeater.cc:464] Registering TS with master...
I20260929 16:42:11.432420 8213 heartbeater.cc:511] Master 127.0.197.125:34675 requested a full tablet report, sending...
I20260929 16:42:11.433748 7555 ts_manager.cc:194] Registered new tserver with Master: af179336d9f44d9ebe6d2205959274e4 (127.0.197.67:36295)
I20260929 16:42:11.433911 8225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:11.378644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46154 (local address 127.0.197.126:33747)
0929 16:42:11.399886 (+ 21242us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:11.399895 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:11.399922 (+ 27us) server_negotiation.cc:407] Connection header received
0929 16:42:11.399984 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:11.399992 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:11.400089 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:11.404294 (+ 4205us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:11.404507 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:11.406100 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:11.419676 (+ 13576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:11.420338 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:11.420500 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:11.424859 (+ 4359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:11.424867 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:11.424873 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:11.424922 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:11.429549 (+ 4627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:11.430564 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:11.430569 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:11.430572 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:11.430767 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:11.431680 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:11.431684 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:11.431688 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:11.432213 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:11.432443 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:11.433265 (+ 822us) server_negotiation.cc:299] Negotiation successful
0929 16:42:11.433510 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20905,"thread_start_us":20152,"threads_started":1}
I20260929 16:42:11.435130 8229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:11.382964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52212 (local address 127.0.197.124:36163)
0929 16:42:11.390730 (+ 7766us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:11.390737 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:11.407920 (+ 17183us) server_negotiation.cc:407] Connection header received
0929 16:42:11.408122 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:11.408128 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:11.408210 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:11.408316 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:11.419608 (+ 11292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:11.420888 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:11.423220 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:11.423885 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:11.423994 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:11.427663 (+ 3669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:11.427668 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:11.427671 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:11.427704 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:11.431300 (+ 3596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:11.432083 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:11.432091 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:11.432097 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:11.432273 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:11.432738 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:11.432746 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:11.432749 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:11.433082 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:11.433351 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:11.434432 (+ 1081us) server_negotiation.cc:299] Negotiation successful
0929 16:42:11.434665 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7576,"thread_start_us":13055,"threads_started":1}
I20260929 16:42:11.435508 8222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:11.381004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57998 (local address 127.0.197.125:34675)
0929 16:42:11.411529 (+ 30525us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:11.411535 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:11.411552 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:42:11.411604 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:11.411609 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:11.411701 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:11.411844 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:11.415648 (+ 3804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:11.416908 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:11.420771 (+ 3863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:11.421452 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:11.422339 (+ 887us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:11.426350 (+ 4011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:11.426357 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:11.426363 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:11.426396 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:11.430407 (+ 4011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:11.431640 (+ 1233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:11.431646 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:11.431651 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:11.431815 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:11.432265 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:11.432272 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:11.432276 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:11.432611 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:11.434716 (+ 2105us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:11.434860 (+ 144us) server_negotiation.cc:299] Negotiation successful
0929 16:42:11.435116 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30369,"thread_start_us":26103,"threads_started":1}
I20260929 16:42:11.438464 6952 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:42:11.446094 6952 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:42:11.486457 8243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:11.468065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:35927 (local address 127.0.0.1:37888)
0929 16:42:11.471624 (+ 3559us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:11.471651 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:11.471839 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:11.472973 (+ 1134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:11.472978 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:11.473011 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:42:11.473385 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:11.473396 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:11.481396 (+ 8000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:11.481402 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:11.483400 (+ 1998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:11.483413 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:11.485532 (+ 2119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:11.485540 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:11.485749 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:11.485821 (+ 72us) client_negotiation.cc:769] Sending connection context
0929 16:42:11.485934 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 16:42:11.486103 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3372,"thread_start_us":5380,"threads_started":1}
I20260929 16:42:11.486500 8242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:11.468301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37888 (local address 127.0.197.65:35927)
0929 16:42:11.472409 (+ 4108us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:11.472424 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:11.472450 (+ 26us) server_negotiation.cc:407] Connection header received
0929 16:42:11.472535 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:11.472541 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:11.472761 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:11.472987 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:42:11.473636 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:11.481113 (+ 7477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:11.483600 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:11.485291 (+ 1691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:11.485437 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:11.485654 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:11.486007 (+ 353us) server_negotiation.cc:299] Negotiation successful
0929 16:42:11.486210 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3943,"thread_start_us":3806,"threads_started":1}
I20260929 16:42:11.489745 8245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:11.473878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.67:36295 (local address 127.0.0.1:33270)
0929 16:42:11.475445 (+ 1567us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:11.475466 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:11.477250 (+ 1784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:11.481639 (+ 4389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:11.481644 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:11.481675 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:42:11.482139 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:11.482153 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:11.484069 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:11.484075 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:11.486193 (+ 2118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:11.486207 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:11.488831 (+ 2624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:11.488843 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:11.489036 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:11.489074 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 16:42:11.489183 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 16:42:11.489393 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1421,"thread_start_us":1174,"threads_started":1}
I20260929 16:42:11.489841 8246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:11.474231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33270 (local address 127.0.197.67:36295)
0929 16:42:11.481023 (+ 6792us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:11.481034 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:11.481073 (+ 39us) server_negotiation.cc:407] Connection header received
0929 16:42:11.481176 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:11.481183 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:11.481428 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:11.481659 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:42:11.482340 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:11.483910 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:11.486812 (+ 2902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:11.488398 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:11.488549 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:11.488807 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:11.489282 (+ 475us) server_negotiation.cc:299] Negotiation successful
0929 16:42:11.489461 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6580,"thread_start_us":7690,"threads_started":1}
I20260929 16:42:11.492295 8241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:11.468776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.66:44381 (local address 127.0.0.1:38880)
0929 16:42:11.473507 (+ 4731us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:11.473528 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:11.473671 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:11.474567 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:11.474575 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:11.474614 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:42:11.475410 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:11.475438 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:11.477207 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:11.477215 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:11.480877 (+ 3662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:11.480896 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:11.491473 (+ 10577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:11.491481 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:11.491687 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:11.491724 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 16:42:11.491821 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 16:42:11.491965 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4561,"thread_start_us":2469,"threads_started":1}
I20260929 16:42:11.492899 8244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:11.469024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38880 (local address 127.0.197.66:44381)
0929 16:42:11.469757 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:11.469768 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:11.473840 (+ 4072us) server_negotiation.cc:407] Connection header received
0929 16:42:11.473962 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:11.473973 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:11.474324 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:11.474549 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:42:11.475640 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:11.477040 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:11.481097 (+ 4057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:11.482695 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:11.482831 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:11.483097 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:11.492098 (+ 9001us) server_negotiation.cc:299] Negotiation successful
0929 16:42:11.492334 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":406,"threads_started":1}
I20260929 16:42:11.490942 7859 tablet_service.cc:1467] Processing CreateTablet for tablet ce63d773f8504318a9a6676c7862b94e (DEFAULT_TABLE table=test-table [id=0c41028707ea438a91b05ccf9ea0dcc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:42:11.494628 7859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce63d773f8504318a9a6676c7862b94e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:42:11.494268 8146 tablet_service.cc:1467] Processing CreateTablet for tablet ce63d773f8504318a9a6676c7862b94e (DEFAULT_TABLE table=test-table [id=0c41028707ea438a91b05ccf9ea0dcc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:42:11.495793 8146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce63d773f8504318a9a6676c7862b94e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:42:11.498857 8005 tablet_service.cc:1467] Processing CreateTablet for tablet ce63d773f8504318a9a6676c7862b94e (DEFAULT_TABLE table=test-table [id=0c41028707ea438a91b05ccf9ea0dcc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:42:11.501251 8005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce63d773f8504318a9a6676c7862b94e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:42:11.512423 8248 tablet_bootstrap.cc:492] T ce63d773f8504318a9a6676c7862b94e P af179336d9f44d9ebe6d2205959274e4: Bootstrap starting.
I20260929 16:42:11.513304 8247 tablet_bootstrap.cc:492] T ce63d773f8504318a9a6676c7862b94e P 52819fca8649475f98ef4c458a2e9a43: Bootstrap starting.
I20260929 16:42:11.518296 8247 tablet_bootstrap.cc:654] T ce63d773f8504318a9a6676c7862b94e P 52819fca8649475f98ef4c458a2e9a43: Neither blocks nor log segments found. Creating new log.
I20260929 16:42:11.518784 8248 tablet_bootstrap.cc:654] T ce63d773f8504318a9a6676c7862b94e P af179336d9f44d9ebe6d2205959274e4: Neither blocks nor log segments found. Creating new log.
I20260929 16:42:11.519861 8249 tablet_bootstrap.cc:492] T ce63d773f8504318a9a6676c7862b94e P 9bed1bd04d834cdd9d75106656df7153: Bootstrap starting.
I20260929 16:42:11.519975 8247 log.cc:824] T ce63d773f8504318a9a6676c7862b94e P 52819fca8649475f98ef4c458a2e9a43: Log is configured to *not* fsync() on all Append() calls
I20260929 16:42:11.520172 8248 log.cc:824] T ce63d773f8504318a9a6676c7862b94e P af179336d9f44d9ebe6d2205959274e4: Log is configured to *not* fsync() on all Append() calls
I20260929 16:42:11.523998 8249 tablet_bootstrap.cc:654] T ce63d773f8504318a9a6676c7862b94e P 9bed1bd04d834cdd9d75106656df7153: Neither blocks nor log segments found. Creating new log.
I20260929 16:42:11.524300 8248 tablet_bootstrap.cc:492] T ce63d773f8504318a9a6676c7862b94e P af179336d9f44d9ebe6d2205959274e4: No bootstrap required, opened a new log
I20260929 16:42:11.524621 8248 ts_tablet_manager.cc:1397] T ce63d773f8504318a9a6676c7862b94e P af179336d9f44d9ebe6d2205959274e4: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260929 16:42:11.525909 8249 log.cc:824] T ce63d773f8504318a9a6676c7862b94e P 9bed1bd04d834cdd9d75106656df7153: Log is configured to *not* fsync() on all Append() calls
I20260929 16:42:11.529665 8247 tablet_bootstrap.cc:492] T ce63d773f8504318a9a6676c7862b94e P 52819fca8649475f98ef4c458a2e9a43: No bootstrap required, opened a new log
I20260929 16:42:11.529665 8249 tablet_bootstrap.cc:492] T ce63d773f8504318a9a6676c7862b94e P 9bed1bd04d834cdd9d75106656df7153: No bootstrap required, opened a new log
I20260929 16:42:11.530014 8249 ts_tablet_manager.cc:1397] T ce63d773f8504318a9a6676c7862b94e P 9bed1bd04d834cdd9d75106656df7153: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20260929 16:42:11.529999 8247 ts_tablet_manager.cc:1397] T ce63d773f8504318a9a6676c7862b94e P 52819fca8649475f98ef4c458a2e9a43: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.010s
I20260929 16:42:11.535094 8248 raft_consensus.cc:348] T ce63d773f8504318a9a6676c7862b94e P af179336d9f44d9ebe6d2205959274e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52819fca8649475f98ef4c458a2e9a43" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 35927 } } peers { permanent_uuid: "9bed1bd04d834cdd9d75106656df7153" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44381 } } peers { permanent_uuid: "af179336d9f44d9ebe6d2205959274e4" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 36295 } }
I20260929 16:42:11.535532 8248 raft_consensus.cc:374] T ce63d773f8504318a9a6676c7862b94e P af179336d9f44d9ebe6d2205959274e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:42:11.535632 8248 raft_consensus.cc:729] T ce63d773f8504318a9a6676c7862b94e P af179336d9f44d9ebe6d2205959274e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af179336d9f44d9ebe6d2205959274e4, State: Initialized, Role: FOLLOWER
I20260929 16:42:11.536310 8248 consensus_queue.cc:260] T ce63d773f8504318a9a6676c7862b94e P af179336d9f44d9ebe6d2205959274e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52819fca8649475f98ef4c458a2e9a43" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 35927 } } peers { permanent_uuid: "9bed1bd04d834cdd9d75106656df7153" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44381 } } peers { permanent_uuid: "af179336d9f44d9ebe6d2205959274e4" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 36295 } }
I20260929 16:42:11.538208 8248 ts_tablet_manager.cc:1428] T ce63d773f8504318a9a6676c7862b94e P af179336d9f44d9ebe6d2205959274e4: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260929 16:42:11.538503 8212 heartbeater.cc:503] Master 127.0.197.126:33747 was elected leader, sending a full tablet report...
I20260929 16:42:11.541350 8247 raft_consensus.cc:348] T ce63d773f8504318a9a6676c7862b94e P 52819fca8649475f98ef4c458a2e9a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52819fca8649475f98ef4c458a2e9a43" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 35927 } } peers { permanent_uuid: "9bed1bd04d834cdd9d75106656df7153" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44381 } } peers { permanent_uuid: "af179336d9f44d9ebe6d2205959274e4" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 36295 } }
I20260929 16:42:11.541350 8249 raft_consensus.cc:348] T ce63d773f8504318a9a6676c7862b94e P 9bed1bd04d834cdd9d75106656df7153 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52819fca8649475f98ef4c458a2e9a43" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 35927 } } peers { permanent_uuid: "9bed1bd04d834cdd9d75106656df7153" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44381 } } peers { permanent_uuid: "af179336d9f44d9ebe6d2205959274e4" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 36295 } }
I20260929 16:42:11.541855 8249 raft_consensus.cc:374] T ce63d773f8504318a9a6676c7862b94e P 9bed1bd04d834cdd9d75106656df7153 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:42:11.541855 8247 raft_consensus.cc:374] T ce63d773f8504318a9a6676c7862b94e P 52819fca8649475f98ef4c458a2e9a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:42:11.541967 8249 raft_consensus.cc:729] T ce63d773f8504318a9a6676c7862b94e P 9bed1bd04d834cdd9d75106656df7153 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bed1bd04d834cdd9d75106656df7153, State: Initialized, Role: FOLLOWER
I20260929 16:42:11.541960 8247 raft_consensus.cc:729] T ce63d773f8504318a9a6676c7862b94e P 52819fca8649475f98ef4c458a2e9a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52819fca8649475f98ef4c458a2e9a43, State: Initialized, Role: FOLLOWER
I20260929 16:42:11.542672 8247 consensus_queue.cc:260] T ce63d773f8504318a9a6676c7862b94e P 52819fca8649475f98ef4c458a2e9a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52819fca8649475f98ef4c458a2e9a43" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 35927 } } peers { permanent_uuid: "9bed1bd04d834cdd9d75106656df7153" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44381 } } peers { permanent_uuid: "af179336d9f44d9ebe6d2205959274e4" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 36295 } }
I20260929 16:42:11.542649 8249 consensus_queue.cc:260] T ce63d773f8504318a9a6676c7862b94e P 9bed1bd04d834cdd9d75106656df7153 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52819fca8649475f98ef4c458a2e9a43" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 35927 } } peers { permanent_uuid: "9bed1bd04d834cdd9d75106656df7153" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44381 } } peers { permanent_uuid: "af179336d9f44d9ebe6d2205959274e4" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 36295 } }
I20260929 16:42:11.544834 8071 heartbeater.cc:503] Master 127.0.197.126:33747 was elected leader, sending a full tablet report...
I20260929 16:42:11.545140 8247 ts_tablet_manager.cc:1428] T ce63d773f8504318a9a6676c7862b94e P 52819fca8649475f98ef4c458a2e9a43: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260929 16:42:11.545327 8249 ts_tablet_manager.cc:1428] T ce63d773f8504318a9a6676c7862b94e P 9bed1bd04d834cdd9d75106656df7153: Time spent starting tablet: real 0.015s user 0.007s sys 0.008s
W20260929 16:42:11.621654 8215 tablet.cc:2369] T ce63d773f8504318a9a6676c7862b94e P af179336d9f44d9ebe6d2205959274e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:42:11.658170 8255 raft_consensus.cc:482] T ce63d773f8504318a9a6676c7862b94e P 9bed1bd04d834cdd9d75106656df7153 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:42:11.658517 8255 raft_consensus.cc:504] T ce63d773f8504318a9a6676c7862b94e P 9bed1bd04d834cdd9d75106656df7153 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52819fca8649475f98ef4c458a2e9a43" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 35927 } } peers { permanent_uuid: "9bed1bd04d834cdd9d75106656df7153" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44381 } } peers { permanent_uuid: "af179336d9f44d9ebe6d2205959274e4" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 36295 } }
I20260929 16:42:11.660394 8255 leader_election.cc:288] T ce63d773f8504318a9a6676c7862b94e P 9bed1bd04d834cdd9d75106656df7153 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52819fca8649475f98ef4c458a2e9a43 (127.0.197.65:35927), af179336d9f44d9ebe6d2205959274e4 (127.0.197.67:36295)
I20260929 16:42:11.667887 8242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:11.660945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:58769 (local address 127.0.197.65:35927)
0929 16:42:11.661148 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:11.661152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:11.661536 (+ 384us) server_negotiation.cc:407] Connection header received
0929 16:42:11.661792 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:11.661803 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:11.661904 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:11.661970 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:42:11.662775 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:11.663993 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:11.666041 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:11.666856 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:11.667089 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:11.667180 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:11.667506 (+ 326us) server_negotiation.cc:299] Negotiation successful
0929 16:42:11.667632 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 16:42:11.667889 8246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:11.661786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:46349 (local address 127.0.197.67:36295)
0929 16:42:11.662014 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:11.662020 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:11.662222 (+ 202us) server_negotiation.cc:407] Connection header received
0929 16:42:11.662464 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:11.662469 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:11.662535 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:11.662640 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:42:11.663445 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:11.664178 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:11.666037 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:11.666842 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:11.667087 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:11.667176 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:11.667491 (+ 315us) server_negotiation.cc:299] Negotiation successful
0929 16:42:11.667622 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 16:42:11.668056 8257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:11.661546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.67:36295 (local address 127.0.197.66:46349)
0929 16:42:11.662115 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:11.662130 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:11.662320 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:11.662654 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:11.662658 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:11.662681 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:42:11.662998 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:11.663059 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:11.664340 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:11.664347 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:11.665857 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:11.665871 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:11.667135 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:11.667143 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:11.667312 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:11.667338 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 16:42:11.667453 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 16:42:11.667600 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"mutex_wait_us":54,"thread_start_us":288,"threads_started":1}
I20260929 16:42:11.668535 8256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:11.660705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:35927 (local address 127.0.197.66:58769)
0929 16:42:11.661393 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:11.661415 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:11.661567 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:11.662167 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:11.662170 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:11.662194 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:42:11.662488 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:11.662590 (+ 102us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:11.664150 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:11.664158 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:11.665860 (+ 1702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:11.665872 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:11.667179 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:11.667186 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:11.667335 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:11.667359 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 16:42:11.667477 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 16:42:11.667618 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"mutex_wait_us":66,"thread_start_us":363,"threads_started":1}
I20260929 16:42:11.669199 8166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce63d773f8504318a9a6676c7862b94e" candidate_uuid: "9bed1bd04d834cdd9d75106656df7153" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af179336d9f44d9ebe6d2205959274e4" is_pre_election: true
I20260929 16:42:11.669528 7879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce63d773f8504318a9a6676c7862b94e" candidate_uuid: "9bed1bd04d834cdd9d75106656df7153" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52819fca8649475f98ef4c458a2e9a43" is_pre_election: true
I20260929 16:42:11.669863 8166 raft_consensus.cc:2454] T ce63d773f8504318a9a6676c7862b94e P af179336d9f44d9ebe6d2205959274e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bed1bd04d834cdd9d75106656df7153 in term 0.
I20260929 16:42:11.670013 7879 raft_consensus.cc:2454] T ce63d773f8504318a9a6676c7862b94e P 52819fca8649475f98ef4c458a2e9a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bed1bd04d834cdd9d75106656df7153 in term 0.
I20260929 16:42:11.670661 7960 leader_election.cc:302] T ce63d773f8504318a9a6676c7862b94e P 9bed1bd04d834cdd9d75106656df7153 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bed1bd04d834cdd9d75106656df7153, af179336d9f44d9ebe6d2205959274e4; no voters:
I20260929 16:42:11.671173 8255 raft_consensus.cc:2781] T ce63d773f8504318a9a6676c7862b94e P 9bed1bd04d834cdd9d75106656df7153 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:42:11.671339 8255 raft_consensus.cc:482] T ce63d773f8504318a9a6676c7862b94e P 9bed1bd04d834cdd9d75106656df7153 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:42:11.671480 8255 raft_consensus.cc:3037] T ce63d773f8504318a9a6676c7862b94e P 9bed1bd04d834cdd9d75106656df7153 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:42:11.673765 8255 raft_consensus.cc:504] T ce63d773f8504318a9a6676c7862b94e P 9bed1bd04d834cdd9d75106656df7153 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52819fca8649475f98ef4c458a2e9a43" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 35927 } } peers { permanent_uuid: "9bed1bd04d834cdd9d75106656df7153" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44381 } } peers { permanent_uuid: "af179336d9f44d9ebe6d2205959274e4" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 36295 } }
I20260929 16:42:11.674418 8255 leader_election.cc:288] T ce63d773f8504318a9a6676c7862b94e P 9bed1bd04d834cdd9d75106656df7153 [CANDIDATE]: Term 1 election: Requested vote from peers 52819fca8649475f98ef4c458a2e9a43 (127.0.197.65:35927), af179336d9f44d9ebe6d2205959274e4 (127.0.197.67:36295)
I20260929 16:42:11.675117 7879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce63d773f8504318a9a6676c7862b94e" candidate_uuid: "9bed1bd04d834cdd9d75106656df7153" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52819fca8649475f98ef4c458a2e9a43"
I20260929 16:42:11.675223 8166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce63d773f8504318a9a6676c7862b94e" candidate_uuid: "9bed1bd04d834cdd9d75106656df7153" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af179336d9f44d9ebe6d2205959274e4"
I20260929 16:42:11.675347 7879 raft_consensus.cc:3037] T ce63d773f8504318a9a6676c7862b94e P 52819fca8649475f98ef4c458a2e9a43 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:42:11.675499 8166 raft_consensus.cc:3037] T ce63d773f8504318a9a6676c7862b94e P af179336d9f44d9ebe6d2205959274e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:42:11.677850 7879 raft_consensus.cc:2454] T ce63d773f8504318a9a6676c7862b94e P 52819fca8649475f98ef4c458a2e9a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bed1bd04d834cdd9d75106656df7153 in term 1.
I20260929 16:42:11.678511 7960 leader_election.cc:302] T ce63d773f8504318a9a6676c7862b94e P 9bed1bd04d834cdd9d75106656df7153 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52819fca8649475f98ef4c458a2e9a43, 9bed1bd04d834cdd9d75106656df7153; no voters:
I20260929 16:42:11.678594 8166 raft_consensus.cc:2454] T ce63d773f8504318a9a6676c7862b94e P af179336d9f44d9ebe6d2205959274e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bed1bd04d834cdd9d75106656df7153 in term 1.
I20260929 16:42:11.678891 8255 raft_consensus.cc:2781] T ce63d773f8504318a9a6676c7862b94e P 9bed1bd04d834cdd9d75106656df7153 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:42:11.679500 8255 raft_consensus.cc:686] T ce63d773f8504318a9a6676c7862b94e P 9bed1bd04d834cdd9d75106656df7153 [term 1 LEADER]: Becoming Leader. State: Replica: 9bed1bd04d834cdd9d75106656df7153, State: Running, Role: LEADER
I20260929 16:42:11.680281 8255 consensus_queue.cc:237] T ce63d773f8504318a9a6676c7862b94e P 9bed1bd04d834cdd9d75106656df7153 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52819fca8649475f98ef4c458a2e9a43" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 35927 } } peers { permanent_uuid: "9bed1bd04d834cdd9d75106656df7153" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44381 } } peers { permanent_uuid: "af179336d9f44d9ebe6d2205959274e4" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 36295 } }
I20260929 16:42:11.685735 6952 catalog_manager.cc:5495] T ce63d773f8504318a9a6676c7862b94e P 9bed1bd04d834cdd9d75106656df7153 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9bed1bd04d834cdd9d75106656df7153 (127.0.197.66). New cstate: current_term: 1 leader_uuid: "9bed1bd04d834cdd9d75106656df7153" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52819fca8649475f98ef4c458a2e9a43" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 35927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bed1bd04d834cdd9d75106656df7153" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af179336d9f44d9ebe6d2205959274e4" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 36295 } health_report { overall_health: UNKNOWN } } }
W20260929 16:42:11.731719 8079 tablet.cc:2369] T ce63d773f8504318a9a6676c7862b94e P 9bed1bd04d834cdd9d75106656df7153: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:42:11.738821 8244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:11.733162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38894 (local address 127.0.197.66:44381)
0929 16:42:11.733400 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:11.733407 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:11.733526 (+ 119us) server_negotiation.cc:407] Connection header received
0929 16:42:11.733790 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:11.733794 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:11.733872 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:11.734031 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:42:11.734417 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:11.735130 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:11.736606 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:11.737017 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:11.737161 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:11.737677 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:42:11.738221 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:42:11.738388 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:11.738445 (+ 57us) server_negotiation.cc:299] Negotiation successful
0929 16:42:11.738528 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260929 16:42:11.748443 8166 raft_consensus.cc:1261] T ce63d773f8504318a9a6676c7862b94e P af179336d9f44d9ebe6d2205959274e4 [term 1 FOLLOWER]: Refusing update from remote peer 9bed1bd04d834cdd9d75106656df7153: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:42:11.748617 7879 raft_consensus.cc:1261] T ce63d773f8504318a9a6676c7862b94e P 52819fca8649475f98ef4c458a2e9a43 [term 1 FOLLOWER]: Refusing update from remote peer 9bed1bd04d834cdd9d75106656df7153: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:42:11.749534 8258 consensus_queue.cc:1035] T ce63d773f8504318a9a6676c7862b94e P 9bed1bd04d834cdd9d75106656df7153 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af179336d9f44d9ebe6d2205959274e4" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 36295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:42:11.749913 8255 consensus_queue.cc:1035] T ce63d773f8504318a9a6676c7862b94e P 9bed1bd04d834cdd9d75106656df7153 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52819fca8649475f98ef4c458a2e9a43" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 35927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:42:11.770567 8263 mvcc.cc:205] Tried to move back new op lower bound from 7334707739631001600 to 7334707739366596608. Current Snapshot: MvccSnapshot[applied={T|T < 7334707739631001600}]
I20260929 16:42:11.789695 6951 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:42:11.789984 6951 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:42:11.797039 6951 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ab1f8d56bd194fb2b0c5bd8cc6c2993f: Sending DeleteTablet for 3 replicas of tablet ce63d773f8504318a9a6676c7862b94e
I20260929 16:42:11.798748 8005 tablet_service.cc:1514] Processing DeleteTablet for tablet ce63d773f8504318a9a6676c7862b94e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:42:11 UTC) from {username='oryx', principal='oryx/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:38880
I20260929 16:42:11.800235 8272 tablet_replica.cc:330] T ce63d773f8504318a9a6676c7862b94e P 9bed1bd04d834cdd9d75106656df7153: stopping tablet replica
I20260929 16:42:11.800596 7859 tablet_service.cc:1514] Processing DeleteTablet for tablet ce63d773f8504318a9a6676c7862b94e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:42:11 UTC) from {username='oryx', principal='oryx/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:37888
I20260929 16:42:11.801261 8272 raft_consensus.cc:2229] T ce63d773f8504318a9a6676c7862b94e P 9bed1bd04d834cdd9d75106656df7153 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:42:11.801788 8146 tablet_service.cc:1514] Processing DeleteTablet for tablet ce63d773f8504318a9a6676c7862b94e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:42:11 UTC) from {username='oryx', principal='oryx/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:33270
I20260929 16:42:11.802884 8272 raft_consensus.cc:2258] T ce63d773f8504318a9a6676c7862b94e P 9bed1bd04d834cdd9d75106656df7153 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:42:11.806183 8272 ts_tablet_manager.cc:1905] T ce63d773f8504318a9a6676c7862b94e P 9bed1bd04d834cdd9d75106656df7153: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:42:11.811530 8275 tablet_replica.cc:330] T ce63d773f8504318a9a6676c7862b94e P af179336d9f44d9ebe6d2205959274e4: stopping tablet replica
I20260929 16:42:11.812053 8275 raft_consensus.cc:2229] T ce63d773f8504318a9a6676c7862b94e P af179336d9f44d9ebe6d2205959274e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:42:11.812654 8275 raft_consensus.cc:2258] T ce63d773f8504318a9a6676c7862b94e P af179336d9f44d9ebe6d2205959274e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:42:11.812815 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 6920
I20260929 16:42:11.813941 6982 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:42:11.814625 8274 tablet_replica.cc:330] T ce63d773f8504318a9a6676c7862b94e P 52819fca8649475f98ef4c458a2e9a43: stopping tablet replica
I20260929 16:42:11.817010 8272 ts_tablet_manager.cc:1918] T ce63d773f8504318a9a6676c7862b94e P 9bed1bd04d834cdd9d75106656df7153: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:42:11.817318 8272 log.cc:1196] T ce63d773f8504318a9a6676c7862b94e P 9bed1bd04d834cdd9d75106656df7153: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-1/wal/wals/ce63d773f8504318a9a6676c7862b94e
I20260929 16:42:11.818092 8272 ts_tablet_manager.cc:1939] T ce63d773f8504318a9a6676c7862b94e P 9bed1bd04d834cdd9d75106656df7153: Deleting consensus metadata
I20260929 16:42:11.820281 8275 ts_tablet_manager.cc:1905] T ce63d773f8504318a9a6676c7862b94e P af179336d9f44d9ebe6d2205959274e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:42:11.821666 8274 raft_consensus.cc:2229] T ce63d773f8504318a9a6676c7862b94e P 52819fca8649475f98ef4c458a2e9a43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:42:11.822506 8274 raft_consensus.cc:2258] T ce63d773f8504318a9a6676c7862b94e P 52819fca8649475f98ef4c458a2e9a43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:42:11.828462 8274 ts_tablet_manager.cc:1905] T ce63d773f8504318a9a6676c7862b94e P 52819fca8649475f98ef4c458a2e9a43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:42:11.837476 8275 ts_tablet_manager.cc:1918] T ce63d773f8504318a9a6676c7862b94e P af179336d9f44d9ebe6d2205959274e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:42:11.837824 8275 log.cc:1196] T ce63d773f8504318a9a6676c7862b94e P af179336d9f44d9ebe6d2205959274e4: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-2/wal/wals/ce63d773f8504318a9a6676c7862b94e
I20260929 16:42:11.838434 8275 ts_tablet_manager.cc:1939] T ce63d773f8504318a9a6676c7862b94e P af179336d9f44d9ebe6d2205959274e4: Deleting consensus metadata
I20260929 16:42:11.842180 8274 ts_tablet_manager.cc:1918] T ce63d773f8504318a9a6676c7862b94e P 52819fca8649475f98ef4c458a2e9a43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:42:11.842882 8274 log.cc:1196] T ce63d773f8504318a9a6676c7862b94e P 52819fca8649475f98ef4c458a2e9a43: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-0/wal/wals/ce63d773f8504318a9a6676c7862b94e
I20260929 16:42:11.843642 8274 ts_tablet_manager.cc:1939] T ce63d773f8504318a9a6676c7862b94e P 52819fca8649475f98ef4c458a2e9a43: Deleting consensus metadata
I20260929 16:42:11.847918 8222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:11.800785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58008 (local address 127.0.197.125:34675)
0929 16:42:11.801112 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:11.801119 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:11.811686 (+ 10567us) server_negotiation.cc:407] Connection header received
0929 16:42:11.814303 (+ 2617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:11.814311 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:11.814405 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:11.814522 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:11.817654 (+ 3132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:11.818899 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:11.827077 (+ 8178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:11.827805 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:11.827966 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:11.839327 (+ 11361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:11.839337 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:11.839344 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:11.839407 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:11.844217 (+ 4810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:11.845352 (+ 1135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:11.845362 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:11.845368 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:11.845590 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:11.846100 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:11.846104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:11.846107 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:11.846522 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:11.846814 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:11.847237 (+ 423us) server_negotiation.cc:299] Negotiation successful
0929 16:42:11.847513 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195}
I20260929 16:42:11.868068 8229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:11.811404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52214 (local address 127.0.197.124:36163)
0929 16:42:11.811622 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:11.811628 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:11.848138 (+ 36510us) server_negotiation.cc:407] Connection header received
0929 16:42:11.848605 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:11.848612 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:11.848735 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:11.848832 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:11.851234 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:11.852375 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:11.854422 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:11.855226 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:11.855367 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:11.860420 (+ 5053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:11.860429 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:11.860435 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:11.860489 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:11.864556 (+ 4067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:11.865600 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:11.865607 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:11.865612 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:11.865778 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:11.866244 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:11.866249 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:11.866253 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:11.866625 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:11.866883 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:11.867270 (+ 387us) server_negotiation.cc:299] Negotiation successful
0929 16:42:11.867519 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 16:42:11.956136 6938 catalog_manager.cc:4841] TS 52819fca8649475f98ef4c458a2e9a43 (127.0.197.65:35927): tablet ce63d773f8504318a9a6676c7862b94e (table test-table [id=0c41028707ea438a91b05ccf9ea0dcc7]) successfully deleted
I20260929 16:42:11.957485 6938 catalog_manager.cc:4841] TS af179336d9f44d9ebe6d2205959274e4 (127.0.197.67:36295): tablet ce63d773f8504318a9a6676c7862b94e (table test-table [id=0c41028707ea438a91b05ccf9ea0dcc7]) successfully deleted
I20260929 16:42:11.957693 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 6920
I20260929 16:42:12.004627 8229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:11.985287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52226 (local address 127.0.197.124:36163)
0929 16:42:11.985560 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:11.985569 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:11.986682 (+ 1113us) server_negotiation.cc:407] Connection header received
0929 16:42:11.987823 (+ 1141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:11.987831 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:11.987921 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:11.988051 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:11.990554 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:11.991726 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:11.993270 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:11.993681 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:11.993835 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:11.997082 (+ 3247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:11.997088 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:11.997092 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:11.997130 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:12.000899 (+ 3769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:12.001854 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:12.001860 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:12.001864 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:12.002040 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:12.002472 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:12.002480 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:12.002485 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:12.002928 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:12.003232 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:12.003963 (+ 731us) server_negotiation.cc:299] Negotiation successful
0929 16:42:12.004200 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 16:42:12.005992 8222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:11.986403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58020 (local address 127.0.197.125:34675)
0929 16:42:11.986677 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:11.986686 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:11.987488 (+ 802us) server_negotiation.cc:407] Connection header received
0929 16:42:11.987917 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:11.987923 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:11.988016 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:11.988116 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:11.991126 (+ 3010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:11.991901 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:11.993943 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:11.994625 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:11.994727 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:11.998351 (+ 3624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:11.998359 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:11.998364 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:11.998412 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:12.001830 (+ 3418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:12.002750 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:12.002755 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:12.002759 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:12.002913 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:12.004055 (+ 1142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:12.004063 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:12.004066 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:12.004499 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:12.004743 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:12.005397 (+ 654us) server_negotiation.cc:299] Negotiation successful
0929 16:42:12.005642 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 16:42:12.776404 8284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:12.775243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:33747 (local address 127.0.197.67:58863)
0929 16:42:12.775911 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:12.776091 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":332,"threads_started":1}
W20260929 16:42:12.776906 8212 heartbeater.cc:644] Failed to heartbeat to 127.0.197.126:33747 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.197.126:33747: connect: Connection refused (error 111)
I20260929 16:42:12.782465 8285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:12.781595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:33747 (local address 127.0.197.66:37483)
0929 16:42:12.782100 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:12.782204 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":287,"threads_started":1}
W20260929 16:42:12.783064 8071 heartbeater.cc:644] Failed to heartbeat to 127.0.197.126:33747 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.197.126:33747: connect: Connection refused (error 111)
I20260929 16:42:12.785404 8286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:12.784622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:33747 (local address 127.0.197.65:50435)
0929 16:42:12.785122 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:12.785246 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":234,"threads_started":1}
W20260929 16:42:12.786012 7925 heartbeater.cc:644] Failed to heartbeat to 127.0.197.126:33747 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.197.126:33747: connect: Connection refused (error 111)
I20260929 16:42:13.298486 8288 raft_consensus.cc:482] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ab1f8d56bd194fb2b0c5bd8cc6c2993f)
I20260929 16:42:13.298632 8287 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ab1f8d56bd194fb2b0c5bd8cc6c2993f)
I20260929 16:42:13.298861 8288 raft_consensus.cc:504] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33747 } } peers { permanent_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "f5404ea6fa1045c792ce56877881c332" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 36163 } attrs { promote: false } }
I20260929 16:42:13.298869 8287 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33747 } } peers { permanent_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "f5404ea6fa1045c792ce56877881c332" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 36163 } attrs { promote: false } }
I20260929 16:42:13.299872 8287 leader_election.cc:288] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab1f8d56bd194fb2b0c5bd8cc6c2993f (127.0.197.126:33747), 489538cf2b1e4ae68c3ae1c94128ff1f (127.0.197.125:34675)
I20260929 16:42:13.301070 8289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:13.299963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:33747 (local address 127.0.0.1:46534)
0929 16:42:13.300495 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:13.300718 (+ 223us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":269,"threads_started":1}
W20260929 16:42:13.302700 7729 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.197.126:33747: connect: Connection refused (error 111)
I20260929 16:42:13.303457 8290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:13.300416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:33747 (local address 127.0.0.1:46550)
0929 16:42:13.302901 (+ 2485us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:13.303134 (+ 233us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2351,"threads_started":1}
I20260929 16:42:13.306744 8288 leader_election.cc:288] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab1f8d56bd194fb2b0c5bd8cc6c2993f (127.0.197.126:33747), f5404ea6fa1045c792ce56877881c332 (127.0.197.124:36163)
W20260929 16:42:13.311729 7540 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.197.126:33747: connect: Connection refused (error 111)
I20260929 16:42:13.314658 8290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:13.300477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:36163 (local address 127.0.0.1:52228)
0929 16:42:13.303779 (+ 3302us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:13.303803 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:13.304137 (+ 334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:13.305287 (+ 1150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:13.305295 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:13.305339 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:42:13.305671 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:13.305683 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:13.307787 (+ 2104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:13.307796 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:13.310778 (+ 2982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:13.310806 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:13.313836 (+ 3030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:13.313843 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:13.313986 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:13.314010 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 16:42:13.314131 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 16:42:13.314300 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3199,"thread_start_us":2158,"threads_started":1}
I20260929 16:42:13.314658 8293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:13.301141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52228 (local address 127.0.197.124:36163)
0929 16:42:13.302199 (+ 1058us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:13.302208 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:13.303917 (+ 1709us) server_negotiation.cc:407] Connection header received
0929 16:42:13.304551 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:13.304565 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:13.304762 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:13.304881 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:42:13.305904 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:13.307502 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:13.311492 (+ 3990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:13.313334 (+ 1842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:13.313516 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:13.313674 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:13.314169 (+ 495us) server_negotiation.cc:299] Negotiation successful
0929 16:42:13.314316 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":920,"thread_start_us":773,"threads_started":1}
I20260929 16:42:13.315116 8291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:13.314257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:33747 (local address 127.0.0.1:46552)
0929 16:42:13.314555 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:13.314817 (+ 262us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
W20260929 16:42:13.315336 7540 leader_election.cc:334] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ab1f8d56bd194fb2b0c5bd8cc6c2993f (127.0.197.126:33747): Network error: Client connection negotiation failed: client connection to 127.0.197.126:33747: connect: Connection refused (error 111)
I20260929 16:42:13.315611 7754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f5404ea6fa1045c792ce56877881c332" is_pre_election: true
I20260929 16:42:13.315842 8292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:13.315275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:33747 (local address 127.0.0.1:46556)
0929 16:42:13.315464 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:13.315618 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 16:42:13.316049 7754 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 489538cf2b1e4ae68c3ae1c94128ff1f in term 1.
W20260929 16:42:13.316164 7729 leader_election.cc:334] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ab1f8d56bd194fb2b0c5bd8cc6c2993f (127.0.197.126:33747): Network error: Client connection negotiation failed: client connection to 127.0.197.126:33747: connect: Connection refused (error 111)
I20260929 16:42:13.316635 7541 leader_election.cc:302] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 489538cf2b1e4ae68c3ae1c94128ff1f, f5404ea6fa1045c792ce56877881c332; no voters: ab1f8d56bd194fb2b0c5bd8cc6c2993f
I20260929 16:42:13.317071 8288 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 16:42:13.317210 8288 raft_consensus.cc:482] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f [term 1 FOLLOWER]: Starting leader election (detected failure of leader ab1f8d56bd194fb2b0c5bd8cc6c2993f)
I20260929 16:42:13.317356 8288 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f [term 1 FOLLOWER]: Advancing to term 2
I20260929 16:42:13.319810 8288 raft_consensus.cc:504] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33747 } } peers { permanent_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "f5404ea6fa1045c792ce56877881c332" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 36163 } attrs { promote: false } }
I20260929 16:42:13.320276 8289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:13.300596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.125:34675 (local address 127.0.0.1:58034)
0929 16:42:13.301438 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:13.301464 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:13.301631 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:13.313435 (+ 11804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:13.313442 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:13.313513 (+ 71us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:42:13.313894 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:13.313908 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:13.316284 (+ 2376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:13.316291 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:13.317999 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:13.318015 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:13.319447 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:13.319453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:13.319642 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:13.319669 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 16:42:13.319792 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 16:42:13.319950 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"thread_start_us":1757,"threads_started":1}
I20260929 16:42:13.320377 8294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:13.302925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58034 (local address 127.0.197.125:34675)
0929 16:42:13.312820 (+ 9895us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:13.312830 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:13.312854 (+ 24us) server_negotiation.cc:407] Connection header received
0929 16:42:13.312934 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:13.312940 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:13.313031 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:13.314255 (+ 1224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:42:13.314493 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:13.316082 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:13.318210 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:13.319235 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:13.319391 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:13.319516 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:13.319834 (+ 318us) server_negotiation.cc:299] Negotiation successful
0929 16:42:13.319974 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9729,"thread_start_us":9519,"threads_started":1}
I20260929 16:42:13.320634 8288 leader_election.cc:288] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f [CANDIDATE]: Term 2 election: Requested vote from peers ab1f8d56bd194fb2b0c5bd8cc6c2993f (127.0.197.126:33747), f5404ea6fa1045c792ce56877881c332 (127.0.197.124:36163)
I20260929 16:42:13.321290 8291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:13.320828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:33747 (local address 127.0.0.1:46570)
0929 16:42:13.321026 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:13.321121 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 16:42:13.321157 7754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f5404ea6fa1045c792ce56877881c332"
I20260929 16:42:13.321187 7565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f5404ea6fa1045c792ce56877881c332" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" is_pre_election: true
I20260929 16:42:13.321440 7754 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332 [term 1 FOLLOWER]: Advancing to term 2
I20260929 16:42:13.321563 7565 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f5404ea6fa1045c792ce56877881c332 in current term 2: Already voted for candidate 489538cf2b1e4ae68c3ae1c94128ff1f in this term.
I20260929 16:42:13.322144 7729 leader_election.cc:302] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f5404ea6fa1045c792ce56877881c332; no voters: 489538cf2b1e4ae68c3ae1c94128ff1f, ab1f8d56bd194fb2b0c5bd8cc6c2993f
I20260929 16:42:13.322628 8291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:13.322117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:33747 (local address 127.0.0.1:46582)
0929 16:42:13.322371 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:13.322460 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260929 16:42:13.322821 7540 leader_election.cc:334] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ab1f8d56bd194fb2b0c5bd8cc6c2993f (127.0.197.126:33747): Network error: Client connection negotiation failed: client connection to 127.0.197.126:33747: connect: Connection refused (error 111)
I20260929 16:42:13.324164 7754 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 489538cf2b1e4ae68c3ae1c94128ff1f in term 2.
I20260929 16:42:13.324877 8287 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 16:42:13.324970 7541 leader_election.cc:302] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 489538cf2b1e4ae68c3ae1c94128ff1f, f5404ea6fa1045c792ce56877881c332; no voters: ab1f8d56bd194fb2b0c5bd8cc6c2993f
I20260929 16:42:13.325434 8288 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f [term 2 FOLLOWER]: Leader election won for term 2
I20260929 16:42:13.326038 8288 raft_consensus.cc:686] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f [term 2 LEADER]: Becoming Leader. State: Replica: 489538cf2b1e4ae68c3ae1c94128ff1f, State: Running, Role: LEADER
I20260929 16:42:13.326809 8288 consensus_queue.cc:237] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33747 } } peers { permanent_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "f5404ea6fa1045c792ce56877881c332" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 36163 } attrs { promote: false } }
I20260929 16:42:13.328751 8297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 489538cf2b1e4ae68c3ae1c94128ff1f. Latest consensus state: current_term: 2 leader_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33747 } } peers { permanent_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "f5404ea6fa1045c792ce56877881c332" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 36163 } attrs { promote: false } } }
I20260929 16:42:13.329090 8297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f [sys.catalog]: This master's current role is: LEADER
I20260929 16:42:13.329833 8299 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:42:13.333551 8299 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:42:13.334690 8299 catalog_manager.cc:1259] Loaded cluster ID: f0d8d655a3754a4e8a727c9f26d8474b
I20260929 16:42:13.334823 8299 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:42:13.336119 8299 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:42:13.336949 8299 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f: Loaded TSK: 0
I20260929 16:42:13.337860 8299 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:42:13.350320 7555 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58020:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:42:13.358435 7555 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:42:13.363090 8291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:13.362622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:33747 (local address 127.0.0.1:46596)
0929 16:42:13.362802 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:13.362898 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 16:42:13.363303 7754 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332 [term 2 FOLLOWER]: Refusing update from remote peer 489538cf2b1e4ae68c3ae1c94128ff1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260929 16:42:13.364270 8291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:13.363870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:33747 (local address 127.0.0.1:46602)
0929 16:42:13.364032 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:13.364123 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 16:42:13.364159 8297 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5404ea6fa1045c792ce56877881c332" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 36163 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260929 16:42:13.364624 7540 consensus_peers.cc:487] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f -> Peer ab1f8d56bd194fb2b0c5bd8cc6c2993f (127.0.197.126:33747): Couldn't send request to peer ab1f8d56bd194fb2b0c5bd8cc6c2993f. Status: Network error: Client connection negotiation failed: client connection to 127.0.197.126:33747: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 16:42:13.367522 8287 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 489538cf2b1e4ae68c3ae1c94128ff1f. Latest consensus state: current_term: 2 leader_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33747 } } peers { permanent_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "f5404ea6fa1045c792ce56877881c332" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 36163 } attrs { promote: false } } }
I20260929 16:42:13.367826 8287 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:42:13.370153 8287 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33747 } } peers { permanent_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "f5404ea6fa1045c792ce56877881c332" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 36163 } attrs { promote: false } } }
I20260929 16:42:13.370428 8287 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5404ea6fa1045c792ce56877881c332 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:42:13.375438 8288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ab1f8d56bd194fb2b0c5bd8cc6c2993f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33747 } } peers { permanent_uuid: "489538cf2b1e4ae68c3ae1c94128ff1f" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "f5404ea6fa1045c792ce56877881c332" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 36163 } attrs { promote: false } } }
I20260929 16:42:13.375932 8288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f [sys.catalog]: This master's current role is: LEADER
I20260929 16:42:13.393227 8307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:13.385112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33276 (local address 127.0.197.67:36295)
0929 16:42:13.385548 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:13.385554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:13.386918 (+ 1364us) server_negotiation.cc:407] Connection header received
0929 16:42:13.387035 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:13.387040 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:13.387126 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:13.387228 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:42:13.387664 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:13.388819 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:13.390834 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:13.392106 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:13.392608 (+ 502us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:13.392734 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:13.392787 (+ 53us) server_negotiation.cc:299] Negotiation successful
0929 16:42:13.392929 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":561,"threads_started":1}
I20260929 16:42:13.393464 8291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:13.384043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.66:44381 (local address 127.0.0.1:38902)
0929 16:42:13.384375 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:13.384396 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:13.384564 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:13.385520 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:13.385526 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:13.385552 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:42:13.385903 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:13.385918 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:13.387452 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:13.387458 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:13.390909 (+ 3451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:13.390920 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:13.392713 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:13.392720 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:13.392889 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:13.392919 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 16:42:13.393013 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 16:42:13.393157 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":3}
I20260929 16:42:13.393637 8290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:13.385904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.67:36295 (local address 127.0.0.1:33276)
0929 16:42:13.386067 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:13.386088 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:13.386193 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:13.387268 (+ 1075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:13.387272 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:13.387300 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:42:13.387498 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:13.387513 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:13.388976 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:13.388981 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:13.390677 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:13.390689 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:13.392316 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:13.392322 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:13.392590 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:13.392653 (+ 63us) client_negotiation.cc:769] Sending connection context
0929 16:42:13.392741 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 16:42:13.392908 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":91}
I20260929 16:42:13.393302 8309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:13.386378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37896 (local address 127.0.197.65:35927)
0929 16:42:13.387062 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:13.387071 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:13.387088 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:42:13.387158 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:13.387162 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:13.387260 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:13.387358 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:42:13.388249 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:13.389273 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:13.391322 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:13.392117 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:13.392297 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:13.392418 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:13.392840 (+ 422us) server_negotiation.cc:299] Negotiation successful
0929 16:42:13.392991 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":432,"threads_started":1}
I20260929 16:42:13.394135 8306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:13.384352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38902 (local address 127.0.197.66:44381)
0929 16:42:13.385224 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:13.385229 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:13.385245 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:42:13.385302 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:13.385306 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:13.385363 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:13.385487 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:42:13.386096 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:13.387280 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:13.391115 (+ 3835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:13.392334 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:13.392453 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:13.392561 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:13.393657 (+ 1096us) server_negotiation.cc:299] Negotiation successful
0929 16:42:13.393825 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":758,"thread_start_us":647,"threads_started":1}
I20260929 16:42:13.394520 8146 tablet_service.cc:1467] Processing CreateTablet for tablet 72a3b92a91c24811a4591926fb73412f (DEFAULT_TABLE table=test-table [id=01c9c2684268480095c3678e4c96001e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:42:13.394855 8005 tablet_service.cc:1467] Processing CreateTablet for tablet 72a3b92a91c24811a4591926fb73412f (DEFAULT_TABLE table=test-table [id=01c9c2684268480095c3678e4c96001e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:42:13.394961 8146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72a3b92a91c24811a4591926fb73412f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:42:13.395277 8005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72a3b92a91c24811a4591926fb73412f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:42:13.399886 8310 tablet_bootstrap.cc:492] T 72a3b92a91c24811a4591926fb73412f P af179336d9f44d9ebe6d2205959274e4: Bootstrap starting.
I20260929 16:42:13.400020 8311 tablet_bootstrap.cc:492] T 72a3b92a91c24811a4591926fb73412f P 9bed1bd04d834cdd9d75106656df7153: Bootstrap starting.
I20260929 16:42:13.401166 8308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:13.386262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:35927 (local address 127.0.0.1:37896)
0929 16:42:13.386835 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:13.386856 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:13.386961 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:13.387515 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:13.387520 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:13.387546 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:42:13.388061 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:13.388073 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:13.389435 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:13.389440 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:13.391173 (+ 1733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:13.391185 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:13.392360 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:13.392366 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:13.392590 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:13.392647 (+ 57us) client_negotiation.cc:769] Sending connection context
0929 16:42:13.392742 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 16:42:13.400717 (+ 7975us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":7791,"thread_start_us":306,"threads_started":1}
I20260929 16:42:13.401746 8310 tablet_bootstrap.cc:654] T 72a3b92a91c24811a4591926fb73412f P af179336d9f44d9ebe6d2205959274e4: Neither blocks nor log segments found. Creating new log.
I20260929 16:42:13.402158 8311 tablet_bootstrap.cc:654] T 72a3b92a91c24811a4591926fb73412f P 9bed1bd04d834cdd9d75106656df7153: Neither blocks nor log segments found. Creating new log.
I20260929 16:42:13.402146 7859 tablet_service.cc:1467] Processing CreateTablet for tablet 72a3b92a91c24811a4591926fb73412f (DEFAULT_TABLE table=test-table [id=01c9c2684268480095c3678e4c96001e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:42:13.402648 7859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72a3b92a91c24811a4591926fb73412f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:42:13.404717 8311 tablet_bootstrap.cc:492] T 72a3b92a91c24811a4591926fb73412f P 9bed1bd04d834cdd9d75106656df7153: No bootstrap required, opened a new log
I20260929 16:42:13.404883 8311 ts_tablet_manager.cc:1397] T 72a3b92a91c24811a4591926fb73412f P 9bed1bd04d834cdd9d75106656df7153: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 16:42:13.405364 8310 tablet_bootstrap.cc:492] T 72a3b92a91c24811a4591926fb73412f P af179336d9f44d9ebe6d2205959274e4: No bootstrap required, opened a new log
I20260929 16:42:13.405524 8310 ts_tablet_manager.cc:1397] T 72a3b92a91c24811a4591926fb73412f P af179336d9f44d9ebe6d2205959274e4: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260929 16:42:13.405714 8311 raft_consensus.cc:348] T 72a3b92a91c24811a4591926fb73412f P 9bed1bd04d834cdd9d75106656df7153 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bed1bd04d834cdd9d75106656df7153" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44381 } } peers { permanent_uuid: "af179336d9f44d9ebe6d2205959274e4" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 36295 } } peers { permanent_uuid: "52819fca8649475f98ef4c458a2e9a43" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 35927 } }
I20260929 16:42:13.405957 8311 raft_consensus.cc:374] T 72a3b92a91c24811a4591926fb73412f P 9bed1bd04d834cdd9d75106656df7153 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:42:13.406059 8311 raft_consensus.cc:729] T 72a3b92a91c24811a4591926fb73412f P 9bed1bd04d834cdd9d75106656df7153 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bed1bd04d834cdd9d75106656df7153, State: Initialized, Role: FOLLOWER
I20260929 16:42:13.406311 8310 raft_consensus.cc:348] T 72a3b92a91c24811a4591926fb73412f P af179336d9f44d9ebe6d2205959274e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bed1bd04d834cdd9d75106656df7153" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44381 } } peers { permanent_uuid: "af179336d9f44d9ebe6d2205959274e4" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 36295 } } peers { permanent_uuid: "52819fca8649475f98ef4c458a2e9a43" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 35927 } }
I20260929 16:42:13.406534 8310 raft_consensus.cc:374] T 72a3b92a91c24811a4591926fb73412f P af179336d9f44d9ebe6d2205959274e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:42:13.406435 8311 consensus_queue.cc:260] T 72a3b92a91c24811a4591926fb73412f P 9bed1bd04d834cdd9d75106656df7153 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bed1bd04d834cdd9d75106656df7153" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44381 } } peers { permanent_uuid: "af179336d9f44d9ebe6d2205959274e4" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 36295 } } peers { permanent_uuid: "52819fca8649475f98ef4c458a2e9a43" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 35927 } }
I20260929 16:42:13.406631 8310 raft_consensus.cc:729] T 72a3b92a91c24811a4591926fb73412f P af179336d9f44d9ebe6d2205959274e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af179336d9f44d9ebe6d2205959274e4, State: Initialized, Role: FOLLOWER
I20260929 16:42:13.406869 8310 consensus_queue.cc:260] T 72a3b92a91c24811a4591926fb73412f P af179336d9f44d9ebe6d2205959274e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bed1bd04d834cdd9d75106656df7153" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44381 } } peers { permanent_uuid: "af179336d9f44d9ebe6d2205959274e4" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 36295 } } peers { permanent_uuid: "52819fca8649475f98ef4c458a2e9a43" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 35927 } }
I20260929 16:42:13.407344 8311 ts_tablet_manager.cc:1428] T 72a3b92a91c24811a4591926fb73412f P 9bed1bd04d834cdd9d75106656df7153: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 16:42:13.414305 8317 tablet_bootstrap.cc:492] T 72a3b92a91c24811a4591926fb73412f P 52819fca8649475f98ef4c458a2e9a43: Bootstrap starting.
I20260929 16:42:13.417166 8317 tablet_bootstrap.cc:654] T 72a3b92a91c24811a4591926fb73412f P 52819fca8649475f98ef4c458a2e9a43: Neither blocks nor log segments found. Creating new log.
I20260929 16:42:13.418813 8310 ts_tablet_manager.cc:1428] T 72a3b92a91c24811a4591926fb73412f P af179336d9f44d9ebe6d2205959274e4: Time spent starting tablet: real 0.013s user 0.004s sys 0.008s
I20260929 16:42:13.422847 8317 tablet_bootstrap.cc:492] T 72a3b92a91c24811a4591926fb73412f P 52819fca8649475f98ef4c458a2e9a43: No bootstrap required, opened a new log
I20260929 16:42:13.423117 8317 ts_tablet_manager.cc:1397] T 72a3b92a91c24811a4591926fb73412f P 52819fca8649475f98ef4c458a2e9a43: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.004s
I20260929 16:42:13.423832 8316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:13.407961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:33747 (local address 127.0.197.66:59507)
0929 16:42:13.423456 (+ 15495us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:13.423612 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15372,"threads_started":1}
I20260929 16:42:13.424285 8317 raft_consensus.cc:348] T 72a3b92a91c24811a4591926fb73412f P 52819fca8649475f98ef4c458a2e9a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bed1bd04d834cdd9d75106656df7153" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44381 } } peers { permanent_uuid: "af179336d9f44d9ebe6d2205959274e4" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 36295 } } peers { permanent_uuid: "52819fca8649475f98ef4c458a2e9a43" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 35927 } }
I20260929 16:42:13.424635 8317 raft_consensus.cc:374] T 72a3b92a91c24811a4591926fb73412f P 52819fca8649475f98ef4c458a2e9a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:42:13.424747 8317 raft_consensus.cc:729] T 72a3b92a91c24811a4591926fb73412f P 52819fca8649475f98ef4c458a2e9a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52819fca8649475f98ef4c458a2e9a43, State: Initialized, Role: FOLLOWER
I20260929 16:42:13.425038 8317 consensus_queue.cc:260] T 72a3b92a91c24811a4591926fb73412f P 52819fca8649475f98ef4c458a2e9a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bed1bd04d834cdd9d75106656df7153" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44381 } } peers { permanent_uuid: "af179336d9f44d9ebe6d2205959274e4" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 36295 } } peers { permanent_uuid: "52819fca8649475f98ef4c458a2e9a43" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 35927 } }
I20260929 16:42:13.425392 8319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:13.419619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:33747 (local address 127.0.197.67:37769)
0929 16:42:13.424835 (+ 5216us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:13.424967 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5010,"thread_start_us":5473,"threads_started":1}
I20260929 16:42:13.426374 8316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:13.424960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:33747 (local address 127.0.197.66:57663)
0929 16:42:13.425951 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:13.426054 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":875}
I20260929 16:42:13.427879 8213 heartbeater.cc:503] Master 127.0.197.125:34675 was elected leader, sending a full tablet report...
I20260929 16:42:13.427986 8076 heartbeater.cc:503] Master 127.0.197.125:34675 was elected leader, sending a full tablet report...
I20260929 16:42:13.429399 8317 ts_tablet_manager.cc:1428] T 72a3b92a91c24811a4591926fb73412f P 52819fca8649475f98ef4c458a2e9a43: Time spent starting tablet: real 0.006s user 0.003s sys 0.001s
I20260929 16:42:13.431562 8321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:13.427614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:33747 (local address 127.0.197.65:54069)
0929 16:42:13.430411 (+ 2797us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:13.430582 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2590,"thread_start_us":2278,"threads_started":1}
I20260929 16:42:13.432219 7926 heartbeater.cc:503] Master 127.0.197.125:34675 was elected leader, sending a full tablet report...
I20260929 16:42:13.432399 8319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:13.431972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:33747 (local address 127.0.197.67:58529)
0929 16:42:13.432133 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:13.432235 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 16:42:13.433372 8321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:13.432924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:33747 (local address 127.0.197.65:35399)
0929 16:42:13.433142 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:13.433229 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260929 16:42:13.533763 7927 tablet.cc:2369] T 72a3b92a91c24811a4591926fb73412f P 52819fca8649475f98ef4c458a2e9a43: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:42:13.565162 8315 raft_consensus.cc:482] T 72a3b92a91c24811a4591926fb73412f P af179336d9f44d9ebe6d2205959274e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:42:13.565503 8315 raft_consensus.cc:504] T 72a3b92a91c24811a4591926fb73412f P af179336d9f44d9ebe6d2205959274e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bed1bd04d834cdd9d75106656df7153" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44381 } } peers { permanent_uuid: "af179336d9f44d9ebe6d2205959274e4" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 36295 } } peers { permanent_uuid: "52819fca8649475f98ef4c458a2e9a43" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 35927 } }
I20260929 16:42:13.567157 8315 leader_election.cc:288] T 72a3b92a91c24811a4591926fb73412f P af179336d9f44d9ebe6d2205959274e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bed1bd04d834cdd9d75106656df7153 (127.0.197.66:44381), 52819fca8649475f98ef4c458a2e9a43 (127.0.197.65:35927)
I20260929 16:42:13.575559 8306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:13.567484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:42541 (local address 127.0.197.66:44381)
0929 16:42:13.567693 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:13.567700 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:13.567794 (+ 94us) server_negotiation.cc:407] Connection header received
0929 16:42:13.568069 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:13.568078 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:13.568186 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:13.568295 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:42:13.568842 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:13.569778 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:13.572759 (+ 2981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:13.574217 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:13.574437 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:13.574534 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:13.575126 (+ 592us) server_negotiation.cc:299] Negotiation successful
0929 16:42:13.575266 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 16:42:13.575551 8309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:13.567756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:48681 (local address 127.0.197.65:35927)
0929 16:42:13.567997 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:13.568004 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:13.568554 (+ 550us) server_negotiation.cc:407] Connection header received
0929 16:42:13.568703 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:13.568707 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:13.568786 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:13.568883 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:42:13.569408 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:13.570280 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:13.572977 (+ 2697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:13.574362 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:13.574524 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:13.574619 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:13.575095 (+ 476us) server_negotiation.cc:299] Negotiation successful
0929 16:42:13.575252 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 16:42:13.575913 8319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:13.567467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.66:44381 (local address 127.0.197.67:42541)
0929 16:42:13.567648 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:13.567670 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:13.567886 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:13.568342 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:13.568348 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:13.568377 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:42:13.568700 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:13.568713 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:13.569960 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:13.569965 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:13.572531 (+ 2566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:13.572551 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:13.574491 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:13.574501 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:13.574797 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:13.574913 (+ 116us) client_negotiation.cc:769] Sending connection context
0929 16:42:13.575068 (+ 155us) client_negotiation.cc:240] Negotiation successful
0929 16:42:13.575252 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":229}
I20260929 16:42:13.576120 8322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:13.567481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:35927 (local address 127.0.197.67:48681)
0929 16:42:13.568425 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:13.568447 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:13.568569 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:13.569009 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:13.569013 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:13.569035 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:42:13.569254 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:13.569262 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:13.570433 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:13.570439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:13.572790 (+ 2351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:13.572801 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:13.574560 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:13.574565 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:13.574829 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:13.574935 (+ 106us) client_negotiation.cc:769] Sending connection context
0929 16:42:13.575061 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 16:42:13.575252 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":819,"mutex_wait_us":399,"thread_start_us":295,"threads_started":1}
I20260929 16:42:13.576882 7879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a3b92a91c24811a4591926fb73412f" candidate_uuid: "af179336d9f44d9ebe6d2205959274e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52819fca8649475f98ef4c458a2e9a43" is_pre_election: true
I20260929 16:42:13.577168 7879 raft_consensus.cc:2454] T 72a3b92a91c24811a4591926fb73412f P 52819fca8649475f98ef4c458a2e9a43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af179336d9f44d9ebe6d2205959274e4 in term 0.
I20260929 16:42:13.577126 8025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a3b92a91c24811a4591926fb73412f" candidate_uuid: "af179336d9f44d9ebe6d2205959274e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bed1bd04d834cdd9d75106656df7153" is_pre_election: true
I20260929 16:42:13.577620 8025 raft_consensus.cc:2454] T 72a3b92a91c24811a4591926fb73412f P 9bed1bd04d834cdd9d75106656df7153 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af179336d9f44d9ebe6d2205959274e4 in term 0.
I20260929 16:42:13.577824 8101 leader_election.cc:302] T 72a3b92a91c24811a4591926fb73412f P af179336d9f44d9ebe6d2205959274e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52819fca8649475f98ef4c458a2e9a43, af179336d9f44d9ebe6d2205959274e4; no voters:
I20260929 16:42:13.578320 8315 raft_consensus.cc:2781] T 72a3b92a91c24811a4591926fb73412f P af179336d9f44d9ebe6d2205959274e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:42:13.578455 8315 raft_consensus.cc:482] T 72a3b92a91c24811a4591926fb73412f P af179336d9f44d9ebe6d2205959274e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:42:13.578557 8315 raft_consensus.cc:3037] T 72a3b92a91c24811a4591926fb73412f P af179336d9f44d9ebe6d2205959274e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:42:13.581152 8315 raft_consensus.cc:504] T 72a3b92a91c24811a4591926fb73412f P af179336d9f44d9ebe6d2205959274e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bed1bd04d834cdd9d75106656df7153" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44381 } } peers { permanent_uuid: "af179336d9f44d9ebe6d2205959274e4" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 36295 } } peers { permanent_uuid: "52819fca8649475f98ef4c458a2e9a43" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 35927 } }
I20260929 16:42:13.581970 8315 leader_election.cc:288] T 72a3b92a91c24811a4591926fb73412f P af179336d9f44d9ebe6d2205959274e4 [CANDIDATE]: Term 1 election: Requested vote from peers 9bed1bd04d834cdd9d75106656df7153 (127.0.197.66:44381), 52819fca8649475f98ef4c458a2e9a43 (127.0.197.65:35927)
I20260929 16:42:13.582538 8025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a3b92a91c24811a4591926fb73412f" candidate_uuid: "af179336d9f44d9ebe6d2205959274e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bed1bd04d834cdd9d75106656df7153"
I20260929 16:42:13.582545 7879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a3b92a91c24811a4591926fb73412f" candidate_uuid: "af179336d9f44d9ebe6d2205959274e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52819fca8649475f98ef4c458a2e9a43"
I20260929 16:42:13.582772 8025 raft_consensus.cc:3037] T 72a3b92a91c24811a4591926fb73412f P 9bed1bd04d834cdd9d75106656df7153 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:42:13.582777 7879 raft_consensus.cc:3037] T 72a3b92a91c24811a4591926fb73412f P 52819fca8649475f98ef4c458a2e9a43 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:42:13.585973 7879 raft_consensus.cc:2454] T 72a3b92a91c24811a4591926fb73412f P 52819fca8649475f98ef4c458a2e9a43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af179336d9f44d9ebe6d2205959274e4 in term 1.
I20260929 16:42:13.585973 8025 raft_consensus.cc:2454] T 72a3b92a91c24811a4591926fb73412f P 9bed1bd04d834cdd9d75106656df7153 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af179336d9f44d9ebe6d2205959274e4 in term 1.
I20260929 16:42:13.586503 8101 leader_election.cc:302] T 72a3b92a91c24811a4591926fb73412f P af179336d9f44d9ebe6d2205959274e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52819fca8649475f98ef4c458a2e9a43, af179336d9f44d9ebe6d2205959274e4; no voters:
I20260929 16:42:13.586941 8315 raft_consensus.cc:2781] T 72a3b92a91c24811a4591926fb73412f P af179336d9f44d9ebe6d2205959274e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:42:13.587493 8315 raft_consensus.cc:686] T 72a3b92a91c24811a4591926fb73412f P af179336d9f44d9ebe6d2205959274e4 [term 1 LEADER]: Becoming Leader. State: Replica: af179336d9f44d9ebe6d2205959274e4, State: Running, Role: LEADER
I20260929 16:42:13.587952 8315 consensus_queue.cc:237] T 72a3b92a91c24811a4591926fb73412f P af179336d9f44d9ebe6d2205959274e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bed1bd04d834cdd9d75106656df7153" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44381 } } peers { permanent_uuid: "af179336d9f44d9ebe6d2205959274e4" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 36295 } } peers { permanent_uuid: "52819fca8649475f98ef4c458a2e9a43" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 35927 } }
I20260929 16:42:13.588626 8322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:13.588209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:33747 (local address 127.0.197.67:56165)
0929 16:42:13.588384 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:13.588497 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 16:42:13.591809 7554 catalog_manager.cc:5495] T 72a3b92a91c24811a4591926fb73412f P af179336d9f44d9ebe6d2205959274e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to af179336d9f44d9ebe6d2205959274e4 (127.0.197.67). New cstate: current_term: 1 leader_uuid: "af179336d9f44d9ebe6d2205959274e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bed1bd04d834cdd9d75106656df7153" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af179336d9f44d9ebe6d2205959274e4" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 36295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52819fca8649475f98ef4c458a2e9a43" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 35927 } health_report { overall_health: UNKNOWN } } }
I20260929 16:42:13.642040 8307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:13.636547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33290 (local address 127.0.197.67:36295)
0929 16:42:13.636810 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:13.636817 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:13.636841 (+ 24us) server_negotiation.cc:407] Connection header received
0929 16:42:13.636963 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:13.636967 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:13.637053 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:13.637226 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:42:13.637589 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:13.638323 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:13.639755 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:13.640189 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:13.640339 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:13.640782 (+ 443us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:42:13.641399 (+ 617us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:42:13.641580 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:13.641663 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 16:42:13.641756 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260929 16:42:13.646653 7879 raft_consensus.cc:1261] T 72a3b92a91c24811a4591926fb73412f P 52819fca8649475f98ef4c458a2e9a43 [term 1 FOLLOWER]: Refusing update from remote peer af179336d9f44d9ebe6d2205959274e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:42:13.647673 8025 raft_consensus.cc:1261] T 72a3b92a91c24811a4591926fb73412f P 9bed1bd04d834cdd9d75106656df7153 [term 1 FOLLOWER]: Refusing update from remote peer af179336d9f44d9ebe6d2205959274e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:42:13.648116 8323 consensus_queue.cc:1035] T 72a3b92a91c24811a4591926fb73412f P af179336d9f44d9ebe6d2205959274e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52819fca8649475f98ef4c458a2e9a43" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 35927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:42:13.648823 8315 consensus_queue.cc:1035] T 72a3b92a91c24811a4591926fb73412f P af179336d9f44d9ebe6d2205959274e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bed1bd04d834cdd9d75106656df7153" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:42:13.652294 8321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:13.651765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:33747 (local address 127.0.197.65:37799)
0929 16:42:13.651916 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:13.652030 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260929 16:42:13.652550 8316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:13.652066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:33747 (local address 127.0.197.66:53219)
0929 16:42:13.652271 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:13.652371 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260929 16:42:13.660610 8316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:13.660053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:33747 (local address 127.0.197.66:56897)
0929 16:42:13.660316 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:13.660428 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
I20260929 16:42:13.660912 8321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:13.660258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:33747 (local address 127.0.197.65:46633)
0929 16:42:13.660564 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:13.660691 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157}
I20260929 16:42:13.664826 8322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:13.664124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:33747 (local address 127.0.197.67:50637)
0929 16:42:13.664502 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:13.664623 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227}
I20260929 16:42:13.667829 8322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:13.666842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:33747 (local address 127.0.197.67:43853)
0929 16:42:13.667480 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:13.667593 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":510}
I20260929 16:42:13.671651 8329 mvcc.cc:205] Tried to move back new op lower bound from 7334707747408642048 to 7334707747180761088. Current Snapshot: MvccSnapshot[applied={T|T < 7334707747408642048 or (T in {7334707747408642048})}]
I20260929 16:42:13.681952 7555 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:42:13.682277 7555 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:42:13.690786 7555 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 489538cf2b1e4ae68c3ae1c94128ff1f: Sending DeleteTablet for 3 replicas of tablet 72a3b92a91c24811a4591926fb73412f
I20260929 16:42:13.691879 8005 tablet_service.cc:1514] Processing DeleteTablet for tablet 72a3b92a91c24811a4591926fb73412f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:42:13 UTC) from {username='oryx', principal='oryx/127.0.197.125@KRBTEST.COM'} at 127.0.0.1:38902
I20260929 16:42:13.692615 7859 tablet_service.cc:1514] Processing DeleteTablet for tablet 72a3b92a91c24811a4591926fb73412f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:42:13 UTC) from {username='oryx', principal='oryx/127.0.197.125@KRBTEST.COM'} at 127.0.0.1:37896
I20260929 16:42:13.699821 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 7773
I20260929 16:42:13.699837 8146 tablet_service.cc:1514] Processing DeleteTablet for tablet 72a3b92a91c24811a4591926fb73412f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:42:13 UTC) from {username='oryx', principal='oryx/127.0.197.125@KRBTEST.COM'} at 127.0.0.1:33276
I20260929 16:42:13.718314 8338 tablet_replica.cc:330] T 72a3b92a91c24811a4591926fb73412f P af179336d9f44d9ebe6d2205959274e4: stopping tablet replica
I20260929 16:42:13.718544 8337 tablet_replica.cc:330] T 72a3b92a91c24811a4591926fb73412f P 52819fca8649475f98ef4c458a2e9a43: stopping tablet replica
I20260929 16:42:13.718744 8337 raft_consensus.cc:2229] T 72a3b92a91c24811a4591926fb73412f P 52819fca8649475f98ef4c458a2e9a43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:42:13.718763 8338 raft_consensus.cc:2229] T 72a3b92a91c24811a4591926fb73412f P af179336d9f44d9ebe6d2205959274e4 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:42:13.719110 8337 raft_consensus.cc:2258] T 72a3b92a91c24811a4591926fb73412f P 52819fca8649475f98ef4c458a2e9a43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:42:13.719658 8338 raft_consensus.cc:2258] T 72a3b92a91c24811a4591926fb73412f P af179336d9f44d9ebe6d2205959274e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:42:13.721031 8338 ts_tablet_manager.cc:1905] T 72a3b92a91c24811a4591926fb73412f P af179336d9f44d9ebe6d2205959274e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:42:13.723368 8336 tablet_replica.cc:330] T 72a3b92a91c24811a4591926fb73412f P 9bed1bd04d834cdd9d75106656df7153: stopping tablet replica
I20260929 16:42:13.723680 8336 raft_consensus.cc:2229] T 72a3b92a91c24811a4591926fb73412f P 9bed1bd04d834cdd9d75106656df7153 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:42:13.725435 8336 raft_consensus.cc:2258] T 72a3b92a91c24811a4591926fb73412f P 9bed1bd04d834cdd9d75106656df7153 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:42:13.727851 8336 ts_tablet_manager.cc:1905] T 72a3b92a91c24811a4591926fb73412f P 9bed1bd04d834cdd9d75106656df7153: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:42:13.733937 8338 ts_tablet_manager.cc:1918] T 72a3b92a91c24811a4591926fb73412f P af179336d9f44d9ebe6d2205959274e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:42:13.735472 8336 ts_tablet_manager.cc:1918] T 72a3b92a91c24811a4591926fb73412f P 9bed1bd04d834cdd9d75106656df7153: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:42:13.735630 8336 log.cc:1196] T 72a3b92a91c24811a4591926fb73412f P 9bed1bd04d834cdd9d75106656df7153: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-1/wal/wals/72a3b92a91c24811a4591926fb73412f
I20260929 16:42:13.735829 8337 ts_tablet_manager.cc:1905] T 72a3b92a91c24811a4591926fb73412f P 52819fca8649475f98ef4c458a2e9a43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:42:13.736097 8336 ts_tablet_manager.cc:1939] T 72a3b92a91c24811a4591926fb73412f P 9bed1bd04d834cdd9d75106656df7153: Deleting consensus metadata
I20260929 16:42:13.737187 8338 log.cc:1196] T 72a3b92a91c24811a4591926fb73412f P af179336d9f44d9ebe6d2205959274e4: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-2/wal/wals/72a3b92a91c24811a4591926fb73412f
I20260929 16:42:13.737421 7539 catalog_manager.cc:4841] TS 9bed1bd04d834cdd9d75106656df7153 (127.0.197.66:44381): tablet 72a3b92a91c24811a4591926fb73412f (table test-table [id=01c9c2684268480095c3678e4c96001e]) successfully deleted
I20260929 16:42:13.737665 8338 ts_tablet_manager.cc:1939] T 72a3b92a91c24811a4591926fb73412f P af179336d9f44d9ebe6d2205959274e4: Deleting consensus metadata
I20260929 16:42:13.738855 7541 catalog_manager.cc:4841] TS af179336d9f44d9ebe6d2205959274e4 (127.0.197.67:36295): tablet 72a3b92a91c24811a4591926fb73412f (table test-table [id=01c9c2684268480095c3678e4c96001e]) successfully deleted
I20260929 16:42:13.742175 8309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:13.701079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37898 (local address 127.0.197.65:35927)
0929 16:42:13.701291 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:13.701299 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:13.717739 (+ 16440us) server_negotiation.cc:407] Connection header received
0929 16:42:13.717969 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:13.717976 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:13.718085 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:13.718226 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:13.721170 (+ 2944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:13.722409 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:13.724362 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:13.725052 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:13.725193 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:13.730125 (+ 4932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:13.730151 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:13.730173 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:13.730225 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:13.734917 (+ 4692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:13.738825 (+ 3908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:13.738838 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:13.738842 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:13.739097 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:13.739688 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:13.739694 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:13.739698 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:13.740301 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:13.740655 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:13.741371 (+ 716us) server_negotiation.cc:299] Negotiation successful
0929 16:42:13.741673 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 16:42:13.743873 7919 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:42:13.844209 8337 ts_tablet_manager.cc:1918] T 72a3b92a91c24811a4591926fb73412f P 52819fca8649475f98ef4c458a2e9a43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:42:13.844477 8337 log.cc:1196] T 72a3b92a91c24811a4591926fb73412f P 52819fca8649475f98ef4c458a2e9a43: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790700068255611-789-0/minicluster-data/ts-0/wal/wals/72a3b92a91c24811a4591926fb73412f
I20260929 16:42:13.844965 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 7773
I20260929 16:42:13.845041 8337 ts_tablet_manager.cc:1939] T 72a3b92a91c24811a4591926fb73412f P 52819fca8649475f98ef4c458a2e9a43: Deleting consensus metadata
I20260929 16:42:13.861014 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 7936
I20260929 16:42:13.861825 8308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:13.861343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:35927 (local address 127.0.0.1:37902)
0929 16:42:13.861529 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:13.861628 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.65:35927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 16:42:13.862138 7541 catalog_manager.cc:4584] TS 52819fca8649475f98ef4c458a2e9a43 (127.0.197.65:35927): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 72a3b92a91c24811a4591926fb73412f: Network error: Client connection negotiation failed: client connection to 127.0.197.65:35927: connect: Connection refused (error 111)
I20260929 16:42:13.880288 8306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:13.862428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38912 (local address 127.0.197.66:44381)
0929 16:42:13.862596 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:13.862601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:13.862620 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:42:13.862929 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:13.862934 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:13.863136 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:13.863239 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:13.865042 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:13.865831 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:13.868006 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:13.868559 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:13.868694 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:13.872856 (+ 4162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:13.872874 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:13.872894 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:13.872943 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:13.876621 (+ 3678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:13.877704 (+ 1083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:13.877722 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:13.877725 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:13.877892 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:13.878492 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:13.878496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:13.878498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:13.879106 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:13.879364 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:13.879714 (+ 350us) server_negotiation.cc:299] Negotiation successful
0929 16:42:13.879963 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 16:42:13.881137 8065 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:42:13.916473 8308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:13.915941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:33747 (local address 127.0.0.1:46612)
0929 16:42:13.916168 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:13.916279 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 16:42:13.917673 8308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:13.917264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:33747 (local address 127.0.0.1:46620)
0929 16:42:13.917431 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:13.917521 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:33747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 16:42:13.922678 8308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:13.922192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:35927 (local address 127.0.0.1:37914)
0929 16:42:13.922404 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:13.922504 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.65:35927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260929 16:42:13.923947 8308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:13.923529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:35927 (local address 127.0.0.1:37920)
0929 16:42:13.923714 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:13.923801 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.65:35927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 16:42:13.992794 8308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:13.992232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:35927 (local address 127.0.0.1:37928)
0929 16:42:13.992485 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:13.992593 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.65:35927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260929 16:42:13.994032 8308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:13.993595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:35927 (local address 127.0.0.1:37936)
0929 16:42:13.993781 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:13.993877 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.65:35927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260929 16:42:14.005919 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 7936
I20260929 16:42:14.029997 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 8083
I20260929 16:42:14.054107 8307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:14.030954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33306 (local address 127.0.197.67:36295)
0929 16:42:14.031146 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:14.031151 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:14.031169 (+ 18us) server_negotiation.cc:407] Connection header received
0929 16:42:14.031297 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:14.031301 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:14.031379 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:14.031460 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:14.033393 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:14.034267 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:14.036289 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:14.036756 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:14.037000 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:14.047477 (+ 10477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:14.047494 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:14.047513 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:14.047568 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:14.050899 (+ 3331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:14.051925 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:14.051933 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:14.051935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:14.052086 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:14.052538 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:14.052542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:14.052544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:14.052943 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:14.053185 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:14.053564 (+ 379us) server_negotiation.cc:299] Negotiation successful
0929 16:42:14.053746 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 16:42:14.054899 8206 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:42:14.096020 8308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:14.095550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:35927 (local address 127.0.0.1:37942)
0929 16:42:14.095766 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:14.095848 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.65:35927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 16:42:14.097090 8308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:14.096745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:35927 (local address 127.0.0.1:37954)
0929 16:42:14.096893 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:14.096973 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.65:35927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 16:42:14.187072 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 8083
I20260929 16:42:14.208657 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 7415
I20260929 16:42:14.209797 7585 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:42:14.332639 8308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:14.332047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:35927 (local address 127.0.0.1:42686)
0929 16:42:14.332377 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:14.332455 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.65:35927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234}
I20260929 16:42:14.332733 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 7415
I20260929 16:42:14.354380 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 7519
I20260929 16:42:14.355654 7775 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:42:14.474426 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 7519
2026-09-29T16:42:14Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10317 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:42:14 dist-test-slave-l31f krb5kdc[8347](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:42:14 dist-test-slave-l31f krb5kdc[8347](info): set up 2 sockets
Sep 29 16:42:14 dist-test-slave-l31f krb5kdc[8347](info): commencing operation
krb5kdc: starting...
W20260929 16:42:15.609756 789 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.001s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/krb5kdc/test-user.keytab.
Sep 29 16:42:15 dist-test-slave-l31f krb5kdc[8347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:42:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:42:15Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.197.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.197.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.197.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/krb5kdc/oryx_127.0.197.126.keytab.
Entry for principal oryx/127.0.197.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/krb5kdc/oryx_127.0.197.126.keytab.
I20260929 16:42:15.871449 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:38787
--webserver_interface=127.0.197.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/krb5kdc/oryx_127.0.197.126.keytab
--principal=oryx/127.0.197.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:38567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.126:38787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:42:16.231295 8360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:42:16.231681 8360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:42:16.231868 8360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:42:16.239488 8360 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:42:16.239665 8360 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 16:42:16.239765 8360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:42:16.239863 8360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:42:16.239959 8360 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:42:16.240049 8360 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:42:16.250407 8360 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:38567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.197.126:38787
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/krb5kdc/oryx_127.0.197.126.keytab
--principal=oryx/127.0.197.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:38787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.197.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:42:16.251827 8360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:42:16.253808 8360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:42:16.268563 8365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:16.268595 8368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:16.268886 8366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:42:16.269764 8360 server_base.cc:1034] running on GCE node
I20260929 16:42:16.271080 8360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:42:16.272776 8360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:42:16.274097 8360 hybrid_clock.cc:648] HybridClock initialized: now 1790700136274027 us; error 78 us; skew 500 ppm
I20260929 16:42:16.274510 8360 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:42:16 dist-test-slave-l31f krb5kdc[8347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700136, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.197.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:42:16.278263 8360 init.cc:377] Logged in from keytab as oryx/127.0.197.126@KRBTEST.COM (short username oryx)
I20260929 16:42:16.280409 8360 webserver.cc:460] Webserver started at http://127.0.197.126:39387/ using document root <none> and password file <none>
I20260929 16:42:16.281312 8360 fs_manager.cc:359] Metadata directory not provided
I20260929 16:42:16.281477 8360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:42:16.281865 8360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:42:16.285626 8360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/master-0/data/instance:
uuid: "a62329d5a3f14fc2846e3e1a948f8aa9"
format_stamp: "Formatted at 2026-09-29 16:42:16 on dist-test-slave-l31f"
server_key: "7424e987b59394e32b46c0ba2bdeaf81"
server_key_iv: "ca11bad6254749c20c597999d67ede71"
server_key_version: "encryptionkey@0"
I20260929 16:42:16.286777 8360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/master-0/wal/instance:
uuid: "a62329d5a3f14fc2846e3e1a948f8aa9"
format_stamp: "Formatted at 2026-09-29 16:42:16 on dist-test-slave-l31f"
server_key: "7424e987b59394e32b46c0ba2bdeaf81"
server_key_iv: "ca11bad6254749c20c597999d67ede71"
server_key_version: "encryptionkey@0"
I20260929 16:42:16.293193 8360 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 16:42:16.298769 8375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:16.300596 8360 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260929 16:42:16.300865 8360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "a62329d5a3f14fc2846e3e1a948f8aa9"
format_stamp: "Formatted at 2026-09-29 16:42:16 on dist-test-slave-l31f"
server_key: "7424e987b59394e32b46c0ba2bdeaf81"
server_key_iv: "ca11bad6254749c20c597999d67ede71"
server_key_version: "encryptionkey@0"
I20260929 16:42:16.301175 8360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:42:16.325510 8360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:42:16.330935 8360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:42:16.331310 8360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:42:16.358573 8360 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.126:38787
I20260929 16:42:16.358690 8436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.126:38787 every 8 connection(s)
I20260929 16:42:16.360174 8360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:42:16.364327 8437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:42:16.367627 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 8360
I20260929 16:42:16.367900 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/master-0/wal/instance
I20260929 16:42:16.368736 789 external_mini_cluster.cc:1428] Setting key 5e0ec3ad9fb9bec9016cea9001f485ab
I20260929 16:42:16.380621 8437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a62329d5a3f14fc2846e3e1a948f8aa9: Bootstrap starting.
I20260929 16:42:16.385396 8437 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a62329d5a3f14fc2846e3e1a948f8aa9: Neither blocks nor log segments found. Creating new log.
Sep 29 16:42:16 dist-test-slave-l31f krb5kdc[8347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM
I20260929 16:42:16.386792 8437 log.cc:824] T 00000000000000000000000000000000 P a62329d5a3f14fc2846e3e1a948f8aa9: Log is configured to *not* fsync() on all Append() calls
I20260929 16:42:16.390594 8437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a62329d5a3f14fc2846e3e1a948f8aa9: No bootstrap required, opened a new log
I20260929 16:42:16.396829 8440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:16.371284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54156 (local address 127.0.197.126:38787)
0929 16:42:16.372240 (+ 956us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:16.372253 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:16.372299 (+ 46us) server_negotiation.cc:407] Connection header received
0929 16:42:16.374137 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:16.374162 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:16.374608 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:16.375101 (+ 493us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:16.377026 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:16.378953 (+ 1927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:16.381554 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:16.382423 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:16.382608 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:16.387952 (+ 5344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:16.387968 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:16.387995 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:16.388048 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:16.391815 (+ 3767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:16.392826 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:16.392833 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:16.392843 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:16.393012 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:16.393421 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:16.393425 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:16.393427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:16.394071 (+ 644us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:16.394322 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:16.394947 (+ 625us) server_negotiation.cc:299] Negotiation successful
0929 16:42:16.395553 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":724,"thread_start_us":418,"threads_started":1}
I20260929 16:42:16.400410 8437 raft_consensus.cc:348] T 00000000000000000000000000000000 P a62329d5a3f14fc2846e3e1a948f8aa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a62329d5a3f14fc2846e3e1a948f8aa9" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 38787 } }
I20260929 16:42:16.400899 8437 raft_consensus.cc:374] T 00000000000000000000000000000000 P a62329d5a3f14fc2846e3e1a948f8aa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:42:16.401000 8437 raft_consensus.cc:729] T 00000000000000000000000000000000 P a62329d5a3f14fc2846e3e1a948f8aa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a62329d5a3f14fc2846e3e1a948f8aa9, State: Initialized, Role: FOLLOWER
I20260929 16:42:16.401633 8437 consensus_queue.cc:260] T 00000000000000000000000000000000 P a62329d5a3f14fc2846e3e1a948f8aa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a62329d5a3f14fc2846e3e1a948f8aa9" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 38787 } }
I20260929 16:42:16.401853 8437 raft_consensus.cc:388] T 00000000000000000000000000000000 P a62329d5a3f14fc2846e3e1a948f8aa9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:42:16.402009 8437 raft_consensus.cc:482] T 00000000000000000000000000000000 P a62329d5a3f14fc2846e3e1a948f8aa9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:42:16.402179 8437 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a62329d5a3f14fc2846e3e1a948f8aa9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:42:16.404546 8437 raft_consensus.cc:504] T 00000000000000000000000000000000 P a62329d5a3f14fc2846e3e1a948f8aa9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a62329d5a3f14fc2846e3e1a948f8aa9" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 38787 } }
I20260929 16:42:16.405126 8437 leader_election.cc:302] T 00000000000000000000000000000000 P a62329d5a3f14fc2846e3e1a948f8aa9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a62329d5a3f14fc2846e3e1a948f8aa9; no voters:
I20260929 16:42:16.405978 8437 leader_election.cc:288] T 00000000000000000000000000000000 P a62329d5a3f14fc2846e3e1a948f8aa9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:42:16.406147 8442 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a62329d5a3f14fc2846e3e1a948f8aa9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:42:16.407052 8442 raft_consensus.cc:686] T 00000000000000000000000000000000 P a62329d5a3f14fc2846e3e1a948f8aa9 [term 1 LEADER]: Becoming Leader. State: Replica: a62329d5a3f14fc2846e3e1a948f8aa9, State: Running, Role: LEADER
I20260929 16:42:16.407685 8442 consensus_queue.cc:237] T 00000000000000000000000000000000 P a62329d5a3f14fc2846e3e1a948f8aa9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a62329d5a3f14fc2846e3e1a948f8aa9" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 38787 } }
I20260929 16:42:16.408331 8437 sys_catalog.cc:564] T 00000000000000000000000000000000 P a62329d5a3f14fc2846e3e1a948f8aa9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:42:16.410631 8443 sys_catalog.cc:455] T 00000000000000000000000000000000 P a62329d5a3f14fc2846e3e1a948f8aa9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a62329d5a3f14fc2846e3e1a948f8aa9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a62329d5a3f14fc2846e3e1a948f8aa9" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 38787 } } }
I20260929 16:42:16.411147 8443 sys_catalog.cc:458] T 00000000000000000000000000000000 P a62329d5a3f14fc2846e3e1a948f8aa9 [sys.catalog]: This master's current role is: LEADER
I20260929 16:42:16.410682 8444 sys_catalog.cc:455] T 00000000000000000000000000000000 P a62329d5a3f14fc2846e3e1a948f8aa9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a62329d5a3f14fc2846e3e1a948f8aa9. Latest consensus state: current_term: 1 leader_uuid: "a62329d5a3f14fc2846e3e1a948f8aa9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a62329d5a3f14fc2846e3e1a948f8aa9" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 38787 } } }
I20260929 16:42:16.411378 8444 sys_catalog.cc:458] T 00000000000000000000000000000000 P a62329d5a3f14fc2846e3e1a948f8aa9 [sys.catalog]: This master's current role is: LEADER
I20260929 16:42:16.414786 8451 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:42:16.422000 8451 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:42:16.433796 8451 catalog_manager.cc:1347] Generated new cluster ID: ce4918b8da8c463b8f1383d417e1d2c2
I20260929 16:42:16.433938 8451 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:42:16.471899 8451 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:42:16.474052 8451 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:42:16.486764 8451 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a62329d5a3f14fc2846e3e1a948f8aa9: Generated new TSK 0
I20260929 16:42:16.487965 8451 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.197.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.197.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.197.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/krb5kdc/oryx_127.0.197.65.keytab.
Entry for principal oryx/127.0.197.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/krb5kdc/oryx_127.0.197.65.keytab.
I20260929 16:42:16.564338 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.65:0
--local_ip_for_outbound_sockets=127.0.197.65
--webserver_interface=127.0.197.65
--webserver_port=0
--tserver_master_addrs=127.0.197.126:38787
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/krb5kdc/oryx_127.0.197.65.keytab
--principal=oryx/127.0.197.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:38567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:42:16.921593 8463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:42:16.921904 8463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:42:16.922077 8463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:42:16.929663 8463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:42:16.929801 8463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:42:16.929958 8463 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 16:42:16.930073 8463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.65
I20260929 16:42:16.940588 8463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:38567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/krb5kdc/oryx_127.0.197.65.keytab
--principal=oryx/127.0.197.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.197.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.197.126:38787
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.65
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:42:16.942029 8463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:42:16.943534 8463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:42:16.957851 8468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:16.957774 8471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:42:16.958599 8463 server_base.cc:1034] running on GCE node
W20260929 16:42:16.958791 8469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:42:16.959668 8463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:42:16.960757 8463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:42:16.962038 8463 hybrid_clock.cc:648] HybridClock initialized: now 1790700136961955 us; error 57 us; skew 500 ppm
I20260929 16:42:16.962401 8463 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:42:16 dist-test-slave-l31f krb5kdc[8347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700136, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.197.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:42:16.966212 8463 init.cc:377] Logged in from keytab as oryx/127.0.197.65@KRBTEST.COM (short username oryx)
I20260929 16:42:16.968091 8463 webserver.cc:460] Webserver started at http://127.0.197.65:46653/ using document root <none> and password file <none>
I20260929 16:42:16.968943 8463 fs_manager.cc:359] Metadata directory not provided
I20260929 16:42:16.969105 8463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:42:16.969476 8463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:42:16.972941 8463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-0/data/instance:
uuid: "a6d0ac696d0e42ad8808c989facfaee0"
format_stamp: "Formatted at 2026-09-29 16:42:16 on dist-test-slave-l31f"
server_key: "9059bb4b489a8ef1dd3cdaf1c8320509"
server_key_iv: "cb70c2dc1cced38d7ca84730baa15f93"
server_key_version: "encryptionkey@0"
I20260929 16:42:16.973914 8463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-0/wal/instance:
uuid: "a6d0ac696d0e42ad8808c989facfaee0"
format_stamp: "Formatted at 2026-09-29 16:42:16 on dist-test-slave-l31f"
server_key: "9059bb4b489a8ef1dd3cdaf1c8320509"
server_key_iv: "cb70c2dc1cced38d7ca84730baa15f93"
server_key_version: "encryptionkey@0"
I20260929 16:42:16.980311 8463 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260929 16:42:16.985275 8478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:16.987227 8463 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260929 16:42:16.987459 8463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-0/wal
uuid: "a6d0ac696d0e42ad8808c989facfaee0"
format_stamp: "Formatted at 2026-09-29 16:42:16 on dist-test-slave-l31f"
server_key: "9059bb4b489a8ef1dd3cdaf1c8320509"
server_key_iv: "cb70c2dc1cced38d7ca84730baa15f93"
server_key_version: "encryptionkey@0"
I20260929 16:42:16.987696 8463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:42:17.007061 8463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:42:17.012583 8463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:42:17.013339 8463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:42:17.017364 8463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:42:17.017480 8463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:17.017683 8463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:42:17.017759 8463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:42:17 dist-test-slave-l31f krb5kdc[8347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700136, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.197.65@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM
I20260929 16:42:17.043352 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:17.018653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:43853 (local address 127.0.197.126:38787)
0929 16:42:17.019236 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:17.019243 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:17.020703 (+ 1460us) server_negotiation.cc:407] Connection header received
0929 16:42:17.023268 (+ 2565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:17.023277 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:17.023426 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:17.023613 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:17.026336 (+ 2723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:17.027742 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:17.029603 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:17.030061 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:17.030215 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:17.035994 (+ 5779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:17.036003 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:17.036008 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:17.036063 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:17.039494 (+ 3431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:17.040643 (+ 1149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:17.040652 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:17.040658 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:17.040817 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:17.041443 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:17.041450 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:17.041454 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:17.041752 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:17.041912 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:17.042698 (+ 786us) server_negotiation.cc:299] Negotiation successful
0929 16:42:17.042945 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":264,"threads_started":1}
I20260929 16:42:17.045112 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:17.019514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:38787 (local address 127.0.197.65:43853)
0929 16:42:17.020488 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:17.020545 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:17.022722 (+ 2177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:17.023971 (+ 1249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:17.023987 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:17.024903 (+ 916us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:17.026090 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:17.026116 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:17.027968 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:17.027972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:17.029409 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:17.029426 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:17.030354 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:17.030361 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:17.030606 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:17.031501 (+ 895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:17.031549 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:17.035471 (+ 3922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:42:17.039690 (+ 4219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:17.039700 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:17.039732 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:17.040475 (+ 743us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:17.040992 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:17.041000 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:17.041006 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:17.041312 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:17.041887 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:42:17.041897 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:42:17.042317 (+ 420us) client_negotiation.cc:769] Sending connection context
0929 16:42:17.042616 (+ 299us) client_negotiation.cc:240] Negotiation successful
0929 16:42:17.043237 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":419,"threads_started":1}
I20260929 16:42:17.070124 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:17.051587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:59853 (local address 127.0.197.126:38787)
0929 16:42:17.051823 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:17.051832 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:17.052483 (+ 651us) server_negotiation.cc:407] Connection header received
0929 16:42:17.052735 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:17.052742 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:17.052844 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:17.052988 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:17.054536 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:17.055661 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:17.057841 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:17.058727 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:17.059084 (+ 357us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:17.062560 (+ 3476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:17.062566 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:17.062570 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:17.062628 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:17.066268 (+ 3640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:17.067871 (+ 1603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:17.067876 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:17.067878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:17.068076 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:17.068516 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:17.068524 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:17.068527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:17.068874 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:17.069043 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:17.069483 (+ 440us) server_negotiation.cc:299] Negotiation successful
0929 16:42:17.069761 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 16:42:17.070261 8549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:17.051555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:38787 (local address 127.0.197.65:59853)
0929 16:42:17.052313 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:17.052335 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:17.052532 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:17.053018 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:17.053023 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:17.053824 (+ 801us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:17.054349 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:17.054370 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:17.055812 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:17.055821 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:17.057659 (+ 1838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:17.057677 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:17.059517 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:17.059530 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:17.059821 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:17.060953 (+ 1132us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:17.060992 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:17.062072 (+ 1080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:42:17.066470 (+ 4398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:17.066477 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:17.066481 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:17.067627 (+ 1146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:17.068224 (+ 597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:17.068229 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:17.068232 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:17.068396 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:17.069005 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:42:17.069010 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:42:17.069224 (+ 214us) client_negotiation.cc:769] Sending connection context
0929 16:42:17.069385 (+ 161us) client_negotiation.cc:240] Negotiation successful
0929 16:42:17.069771 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":385,"threads_started":1}
I20260929 16:42:17.079768 8463 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.65:44769
I20260929 16:42:17.079823 8598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.65:44769 every 8 connection(s)
I20260929 16:42:17.081122 8463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
I20260929 16:42:17.082085 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 8463
I20260929 16:42:17.082297 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-0/wal/instance
I20260929 16:42:17.083107 789 external_mini_cluster.cc:1428] Setting key ba73916162b0a4dbf716f0dbe2182f23
I20260929 16:42:17.083684 8599 heartbeater.cc:347] Connected to a master server at 127.0.197.126:38787
I20260929 16:42:17.083992 8599 heartbeater.cc:464] Registering TS with master...
I20260929 16:42:17.084820 8599 heartbeater.cc:511] Master 127.0.197.126:38787 requested a full tablet report, sending...
I20260929 16:42:17.087548 8392 ts_manager.cc:194] Registered new tserver with Master: a6d0ac696d0e42ad8808c989facfaee0 (127.0.197.65:44769)
I20260929 16:42:17.089991 8392 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.197.65@KRBTEST.COM'} at 127.0.197.65:43853
WARNING: no policy specified for oryx/127.0.197.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.197.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.197.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/krb5kdc/oryx_127.0.197.66.keytab.
Entry for principal oryx/127.0.197.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/krb5kdc/oryx_127.0.197.66.keytab.
I20260929 16:42:17.140069 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.66:0
--local_ip_for_outbound_sockets=127.0.197.66
--webserver_interface=127.0.197.66
--webserver_port=0
--tserver_master_addrs=127.0.197.126:38787
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/krb5kdc/oryx_127.0.197.66.keytab
--principal=oryx/127.0.197.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:38567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:42:17.492995 8603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:42:17.493311 8603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:42:17.493409 8603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:42:17.500955 8603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:42:17.501103 8603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:42:17.501188 8603 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 16:42:17.501325 8603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.66
I20260929 16:42:17.512718 8603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:38567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/krb5kdc/oryx_127.0.197.66.keytab
--principal=oryx/127.0.197.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.197.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.197.126:38787
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.66
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:42:17.514137 8603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:42:17.515587 8603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:42:17.530273 8611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:17.530309 8609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:17.530238 8608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:42:17.531347 8603 server_base.cc:1034] running on GCE node
I20260929 16:42:17.532068 8603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:42:17.533033 8603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:42:17.534452 8603 hybrid_clock.cc:648] HybridClock initialized: now 1790700137534292 us; error 63 us; skew 500 ppm
I20260929 16:42:17.535280 8603 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:42:17 dist-test-slave-l31f krb5kdc[8347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700137, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.197.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:42:17.539155 8603 init.cc:377] Logged in from keytab as oryx/127.0.197.66@KRBTEST.COM (short username oryx)
I20260929 16:42:17.540846 8603 webserver.cc:460] Webserver started at http://127.0.197.66:44417/ using document root <none> and password file <none>
I20260929 16:42:17.541641 8603 fs_manager.cc:359] Metadata directory not provided
I20260929 16:42:17.541782 8603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:42:17.542138 8603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:42:17.545476 8603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-1/data/instance:
uuid: "2e017281552241b99a107a7afd58b230"
format_stamp: "Formatted at 2026-09-29 16:42:17 on dist-test-slave-l31f"
server_key: "b3c32ab8c1f37817eb3c84a2051db1dd"
server_key_iv: "e507b7c7b29cee736d111e28c0cfbf7c"
server_key_version: "encryptionkey@0"
I20260929 16:42:17.546456 8603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-1/wal/instance:
uuid: "2e017281552241b99a107a7afd58b230"
format_stamp: "Formatted at 2026-09-29 16:42:17 on dist-test-slave-l31f"
server_key: "b3c32ab8c1f37817eb3c84a2051db1dd"
server_key_iv: "e507b7c7b29cee736d111e28c0cfbf7c"
server_key_version: "encryptionkey@0"
I20260929 16:42:17.552673 8603 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260929 16:42:17.556649 8618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:17.558676 8603 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:42:17.558955 8603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-1/wal
uuid: "2e017281552241b99a107a7afd58b230"
format_stamp: "Formatted at 2026-09-29 16:42:17 on dist-test-slave-l31f"
server_key: "b3c32ab8c1f37817eb3c84a2051db1dd"
server_key_iv: "e507b7c7b29cee736d111e28c0cfbf7c"
server_key_version: "encryptionkey@0"
I20260929 16:42:17.559314 8603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:42:17.584753 8603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:42:17.589622 8603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:42:17.589948 8603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:42:17.593758 8603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:42:17.593885 8603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:17.594098 8603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:42:17.594180 8603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:42:17 dist-test-slave-l31f krb5kdc[8347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700137, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.197.66@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM
I20260929 16:42:17.622613 8627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:17.594858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:39633 (local address 127.0.197.126:38787)
0929 16:42:17.595351 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:17.595357 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:17.596830 (+ 1473us) server_negotiation.cc:407] Connection header received
0929 16:42:17.599836 (+ 3006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:17.599842 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:17.599957 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:17.600100 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:17.603798 (+ 3698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:17.604817 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:17.606939 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:17.607945 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:17.608114 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:17.613613 (+ 5499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:17.613623 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:17.613632 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:17.613695 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:17.618116 (+ 4421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:17.619548 (+ 1432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:17.619553 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:17.619555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:17.619761 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:17.620498 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:17.620502 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:17.620504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:17.620820 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:17.620944 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:17.622136 (+ 1192us) server_negotiation.cc:299] Negotiation successful
0929 16:42:17.622333 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":240,"threads_started":1}
I20260929 16:42:17.625105 8628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:17.595642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:38787 (local address 127.0.197.66:39633)
0929 16:42:17.596600 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:17.596654 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:17.599333 (+ 2679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:17.600484 (+ 1151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:17.600503 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:17.601641 (+ 1138us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:17.603276 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:17.603322 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:17.605056 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:17.605062 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:17.606707 (+ 1645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:17.606731 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:17.608241 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:17.608252 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:17.608646 (+ 394us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:17.609758 (+ 1112us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:17.609805 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:17.613288 (+ 3483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:42:17.618347 (+ 5059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:17.618356 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:17.618403 (+ 47us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:17.619245 (+ 842us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:17.619923 (+ 678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:17.619932 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:17.619938 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:17.620356 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:17.620960 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:42:17.620974 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:42:17.621657 (+ 683us) client_negotiation.cc:769] Sending connection context
0929 16:42:17.622059 (+ 402us) client_negotiation.cc:240] Negotiation successful
0929 16:42:17.622848 (+ 789us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":404,"threads_started":1}
I20260929 16:42:17.653141 8627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:17.633230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:58205 (local address 127.0.197.126:38787)
0929 16:42:17.633422 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:17.633427 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:17.636378 (+ 2951us) server_negotiation.cc:407] Connection header received
0929 16:42:17.636616 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:17.636620 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:17.636715 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:17.636817 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:17.638469 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:17.639490 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:17.642610 (+ 3120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:17.643085 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:17.643210 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:17.646254 (+ 3044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:17.646263 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:17.646270 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:17.646320 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:17.649661 (+ 3341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:17.650803 (+ 1142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:17.650807 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:17.650810 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:17.650937 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:17.651488 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:17.651495 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:17.651501 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:17.651931 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:17.652085 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:17.652538 (+ 453us) server_negotiation.cc:299] Negotiation successful
0929 16:42:17.652783 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 16:42:17.653277 8703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:17.632653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:38787 (local address 127.0.197.66:58205)
0929 16:42:17.636156 (+ 3503us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:17.636189 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:17.636436 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:17.636896 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:17.636903 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:17.637709 (+ 806us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:17.638219 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:17.638238 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:17.639715 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:17.639722 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:17.642358 (+ 2636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:17.642385 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:17.643313 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:17.643320 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:17.643539 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:17.644387 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:17.644417 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:17.645975 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:42:17.649859 (+ 3884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:17.649870 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:17.649877 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:17.650639 (+ 762us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:17.651113 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:17.651125 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:17.651132 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:17.651329 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:17.652082 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:42:17.652087 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:42:17.652280 (+ 193us) client_negotiation.cc:769] Sending connection context
0929 16:42:17.652433 (+ 153us) client_negotiation.cc:240] Negotiation successful
0929 16:42:17.652797 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3272,"thread_start_us":3052,"threads_started":1}
I20260929 16:42:17.659987 8603 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.66:42265
I20260929 16:42:17.660012 8738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.66:42265 every 8 connection(s)
I20260929 16:42:17.661425 8603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
I20260929 16:42:17.664018 8739 heartbeater.cc:347] Connected to a master server at 127.0.197.126:38787
I20260929 16:42:17.664276 8739 heartbeater.cc:464] Registering TS with master...
I20260929 16:42:17.664804 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 8603
I20260929 16:42:17.664968 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-1/wal/instance
I20260929 16:42:17.665324 8739 heartbeater.cc:511] Master 127.0.197.126:38787 requested a full tablet report, sending...
I20260929 16:42:17.665483 789 external_mini_cluster.cc:1428] Setting key 99e90092ebd9523dc116ae882f379bf7
I20260929 16:42:17.666992 8391 ts_manager.cc:194] Registered new tserver with Master: 2e017281552241b99a107a7afd58b230 (127.0.197.66:42265)
I20260929 16:42:17.668221 8391 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.197.66@KRBTEST.COM'} at 127.0.197.66:39633
WARNING: no policy specified for oryx/127.0.197.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.197.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.197.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/krb5kdc/oryx_127.0.197.67.keytab.
Entry for principal oryx/127.0.197.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/krb5kdc/oryx_127.0.197.67.keytab.
I20260929 16:42:17.718957 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.67:0
--local_ip_for_outbound_sockets=127.0.197.67
--webserver_interface=127.0.197.67
--webserver_port=0
--tserver_master_addrs=127.0.197.126:38787
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/krb5kdc/oryx_127.0.197.67.keytab
--principal=oryx/127.0.197.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:38567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:42:18.079907 8743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:42:18.080268 8743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:42:18.080444 8743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:42:18.088558 8743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:42:18.088740 8743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:42:18.088897 8743 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 16:42:18.089044 8743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.67
I20260929 16:42:18.093956 8599 heartbeater.cc:503] Master 127.0.197.126:38787 was elected leader, sending a full tablet report...
I20260929 16:42:18.101221 8743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:38567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/krb5kdc/oryx_127.0.197.67.keytab
--principal=oryx/127.0.197.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.197.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.197.126:38787
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.67
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:42:18.102653 8743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:42:18.105046 8743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:42:18.121680 8748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:18.121680 8749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:18.122901 8751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:42:18.128949 8743 server_base.cc:1034] running on GCE node
I20260929 16:42:18.129794 8743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:42:18.130918 8743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:42:18.132206 8743 hybrid_clock.cc:648] HybridClock initialized: now 1790700138132137 us; error 43 us; skew 500 ppm
I20260929 16:42:18.132551 8743 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:42:18 dist-test-slave-l31f krb5kdc[8347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700138, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.197.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:42:18.136678 8743 init.cc:377] Logged in from keytab as oryx/127.0.197.67@KRBTEST.COM (short username oryx)
I20260929 16:42:18.139215 8743 webserver.cc:460] Webserver started at http://127.0.197.67:38193/ using document root <none> and password file <none>
I20260929 16:42:18.140087 8743 fs_manager.cc:359] Metadata directory not provided
I20260929 16:42:18.140246 8743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:42:18.140604 8743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:42:18.145989 8743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-2/data/instance:
uuid: "ef398814e1ff47409b81c7fd8253ef15"
format_stamp: "Formatted at 2026-09-29 16:42:18 on dist-test-slave-l31f"
server_key: "be4c696e8e902b2a33d8847cd75c39c8"
server_key_iv: "87e2114110229d6cf62914b17a8dd86b"
server_key_version: "encryptionkey@0"
I20260929 16:42:18.147145 8743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-2/wal/instance:
uuid: "ef398814e1ff47409b81c7fd8253ef15"
format_stamp: "Formatted at 2026-09-29 16:42:18 on dist-test-slave-l31f"
server_key: "be4c696e8e902b2a33d8847cd75c39c8"
server_key_iv: "87e2114110229d6cf62914b17a8dd86b"
server_key_version: "encryptionkey@0"
I20260929 16:42:18.153415 8743 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260929 16:42:18.158185 8758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:18.160084 8743 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260929 16:42:18.160303 8743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-2/wal
uuid: "ef398814e1ff47409b81c7fd8253ef15"
format_stamp: "Formatted at 2026-09-29 16:42:18 on dist-test-slave-l31f"
server_key: "be4c696e8e902b2a33d8847cd75c39c8"
server_key_iv: "87e2114110229d6cf62914b17a8dd86b"
server_key_version: "encryptionkey@0"
I20260929 16:42:18.160583 8743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:42:18.203758 8743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:42:18.209497 8743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:42:18.209825 8743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:42:18.213390 8743 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:42:18.213526 8743 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:18.213709 8743 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:42:18.213791 8743 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:42:18 dist-test-slave-l31f krb5kdc[8347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700138, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.197.67@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM
I20260929 16:42:18.258473 8767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:18.215369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:52993 (local address 127.0.197.126:38787)
0929 16:42:18.231459 (+ 16090us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:18.231468 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:18.231492 (+ 24us) server_negotiation.cc:407] Connection header received
0929 16:42:18.231577 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:18.231585 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:18.231690 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:18.231830 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:18.234712 (+ 2882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:18.236176 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:18.239978 (+ 3802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:18.240897 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:18.241048 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:18.248546 (+ 7498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:18.248558 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:18.248567 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:18.248626 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:18.253444 (+ 4818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:18.254791 (+ 1347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:18.254801 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:18.254808 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:18.255081 (+ 273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:18.255680 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:18.255689 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:18.255695 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:18.256173 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:18.256351 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:18.257394 (+ 1043us) server_negotiation.cc:299] Negotiation successful
0929 16:42:18.257868 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15925,"thread_start_us":15793,"threads_started":1}
I20260929 16:42:18.259222 8768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:18.215582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:38787 (local address 127.0.197.67:52993)
0929 16:42:18.225229 (+ 9647us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:18.225301 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:18.230533 (+ 5232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:18.232361 (+ 1828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:18.232378 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:18.233325 (+ 947us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:18.234368 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:18.234401 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:18.236464 (+ 2063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:18.236477 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:18.239677 (+ 3200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:18.239711 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:18.241620 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:18.241631 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:18.242095 (+ 464us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:18.243288 (+ 1193us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:18.243323 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:18.248128 (+ 4805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:42:18.253731 (+ 5603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:18.253758 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:18.253779 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:18.254571 (+ 792us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:18.255246 (+ 675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:18.255252 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:18.255256 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:18.255526 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:18.256352 (+ 826us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:42:18.256362 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:42:18.256885 (+ 523us) client_negotiation.cc:769] Sending connection context
0929 16:42:18.257225 (+ 340us) client_negotiation.cc:240] Negotiation successful
0929 16:42:18.257805 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9179,"thread_start_us":11463,"threads_started":1}
I20260929 16:42:18.284816 8767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:18.267395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:50591 (local address 127.0.197.126:38787)
0929 16:42:18.267589 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:18.267595 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:18.268651 (+ 1056us) server_negotiation.cc:407] Connection header received
0929 16:42:18.268884 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:18.268892 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:18.268963 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:18.269060 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:18.270817 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:18.271767 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:18.273819 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:18.274469 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:18.274632 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:18.277819 (+ 3187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:18.277824 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:18.277828 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:18.277864 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:18.281115 (+ 3251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:18.282328 (+ 1213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:18.282337 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:18.282343 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:18.282533 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:18.283152 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:18.283160 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:18.283163 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:18.283515 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:18.283699 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:18.284344 (+ 645us) server_negotiation.cc:299] Negotiation successful
0929 16:42:18.284557 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 16:42:18.285415 8810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:18.267515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:38787 (local address 127.0.197.67:50591)
0929 16:42:18.268455 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:18.268482 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:18.268697 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:18.269171 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:18.269179 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:18.269958 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:18.270452 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:18.270471 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:18.271975 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:18.271980 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:18.273629 (+ 1649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:18.273644 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:18.274726 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:18.274737 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:18.275102 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:18.276293 (+ 1191us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:18.276316 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:18.277545 (+ 1229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:42:18.281309 (+ 3764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:18.281318 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:18.281325 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:18.282122 (+ 797us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:18.282729 (+ 607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:18.282734 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:18.282743 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:18.282950 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:18.283684 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:42:18.283693 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:42:18.283979 (+ 286us) client_negotiation.cc:769] Sending connection context
0929 16:42:18.284210 (+ 231us) client_negotiation.cc:240] Negotiation successful
0929 16:42:18.284742 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"thread_start_us":516,"threads_started":1}
I20260929 16:42:18.300911 8743 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.67:40929
I20260929 16:42:18.300931 8878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.67:40929 every 8 connection(s)
I20260929 16:42:18.302273 8743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
I20260929 16:42:18.304476 8879 heartbeater.cc:347] Connected to a master server at 127.0.197.126:38787
I20260929 16:42:18.304772 8879 heartbeater.cc:464] Registering TS with master...
I20260929 16:42:18.305474 8879 heartbeater.cc:511] Master 127.0.197.126:38787 requested a full tablet report, sending...
I20260929 16:42:18.307081 8391 ts_manager.cc:194] Registered new tserver with Master: ef398814e1ff47409b81c7fd8253ef15 (127.0.197.67:40929)
I20260929 16:42:18.307860 8391 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.197.67@KRBTEST.COM'} at 127.0.197.67:52993
I20260929 16:42:18.308948 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 8743
I20260929 16:42:18.309139 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-2/wal/instance
I20260929 16:42:18.309574 789 external_mini_cluster.cc:1428] Setting key 94664344a4ba010019f2ae56fd7613e2
I20260929 16:42:18.312418 789 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 16:42:18 dist-test-slave-l31f krb5kdc[8347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700138, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 16:42:18 dist-test-slave-l31f krb5kdc[8347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:42:18 dist-test-slave-l31f krb5kdc[8347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:42:18.671168 8739 heartbeater.cc:503] Master 127.0.197.126:38787 was elected leader, sending a full tablet report...
I20260929 16:42:19.311028 8879 heartbeater.cc:503] Master 127.0.197.126:38787 was elected leader, sending a full tablet report...
W20260929 16:42:19.461107 8887 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.099s user 0.006s sys 0.010s
W20260929 16:42:19.462484 8767 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:42:18.354797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54170 (local address 127.0.197.126:38787)
0929 16:42:18.355078 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:18.355085 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:18.355259 (+ 174us) server_negotiation.cc:407] Connection header received
0929 16:42:18.355506 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:18.355511 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:18.355577 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:18.355703 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:18.357088 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:18.357901 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:18.359442 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:18.360135 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:18.360272 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:19.462193 (+1101921us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54170: BlockingRecv error: recv got EOF from 127.0.0.1:54170 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
Sep 29 16:42:19 dist-test-slave-l31f krb5kdc[8347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700138, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM
I20260929 16:42:19.489229 8767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:19.466745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54176 (local address 127.0.197.126:38787)
0929 16:42:19.466990 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:19.466998 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:19.467344 (+ 346us) server_negotiation.cc:407] Connection header received
0929 16:42:19.467580 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:19.467585 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:19.467659 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:19.467811 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:19.469093 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:19.469815 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:19.471145 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:19.471572 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:19.471715 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:19.483335 (+ 11620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:19.483342 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:19.483345 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:19.483391 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:19.486167 (+ 2776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:19.487300 (+ 1133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:19.487305 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:19.487307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:19.487446 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:19.487939 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:19.487943 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:19.487945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:19.488244 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:19.488453 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:19.488807 (+ 354us) server_negotiation.cc:299] Negotiation successful
0929 16:42:19.488994 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 16:42:19.493829 8391 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54176:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:42:19.501108 8391 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:42:19.527994 8901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:19.516818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53382 (local address 127.0.197.67:40929)
0929 16:42:19.520420 (+ 3602us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:19.520432 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:19.520468 (+ 36us) server_negotiation.cc:407] Connection header received
0929 16:42:19.520584 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:19.520596 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:19.520900 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:19.521837 (+ 937us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:42:19.522038 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:19.523488 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:19.525741 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:19.526680 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:19.526852 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:19.527115 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:19.527421 (+ 306us) server_negotiation.cc:299] Negotiation successful
0929 16:42:19.527598 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3426,"thread_start_us":3280,"threads_started":1}
I20260929 16:42:19.528009 8898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:19.516120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.67:40929 (local address 127.0.0.1:53382)
0929 16:42:19.519755 (+ 3635us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:19.519776 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:19.519887 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:19.521169 (+ 1282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:19.521177 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:19.521211 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:42:19.521595 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:19.521607 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:19.523664 (+ 2057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:19.523670 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:19.525574 (+ 1904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:19.525584 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:19.526928 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:19.526935 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:19.527170 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:19.527218 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 16:42:19.527374 (+ 156us) client_negotiation.cc:240] Negotiation successful
0929 16:42:19.527588 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3467,"thread_start_us":360,"threads_started":1}
I20260929 16:42:19.528534 8899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:19.515983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58764 (local address 127.0.197.66:42265)
0929 16:42:19.518198 (+ 2215us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:19.518211 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:19.518249 (+ 38us) server_negotiation.cc:407] Connection header received
0929 16:42:19.518377 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:19.518389 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:19.518732 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:19.518997 (+ 265us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:42:19.520710 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:19.522180 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:19.525116 (+ 2936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:19.526737 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:19.526888 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:19.527112 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:19.527656 (+ 544us) server_negotiation.cc:299] Negotiation successful
0929 16:42:19.527836 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1986,"thread_start_us":1831,"threads_started":1}
I20260929 16:42:19.529328 8900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:19.515826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.66:42265 (local address 127.0.0.1:58764)
0929 16:42:19.516935 (+ 1109us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:19.516986 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:19.517196 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:19.519098 (+ 1902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:19.519110 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:19.519159 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:42:19.519775 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:19.520497 (+ 722us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:19.522381 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:19.522385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:19.524928 (+ 2543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:19.524942 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:19.527276 (+ 2334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:19.527283 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:19.527461 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:19.527493 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 16:42:19.528737 (+ 1244us) client_negotiation.cc:240] Negotiation successful
0929 16:42:19.528880 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":817,"mutex_wait_us":4,"thread_start_us":3623,"threads_started":1}
I20260929 16:42:19.530511 8902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:19.520191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51312 (local address 127.0.197.65:44769)
0929 16:42:19.521483 (+ 1292us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:19.521491 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:19.522215 (+ 724us) server_negotiation.cc:407] Connection header received
0929 16:42:19.522400 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:19.522408 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:19.522634 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:19.522822 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:42:19.523604 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:19.524971 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:19.527479 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:19.528994 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:19.529096 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:19.529347 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:19.530131 (+ 784us) server_negotiation.cc:299] Negotiation successful
0929 16:42:19.530304 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1153,"thread_start_us":1005,"threads_started":1}
I20260929 16:42:19.530517 8903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:19.521549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:44769 (local address 127.0.0.1:51312)
0929 16:42:19.522075 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:19.522095 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:19.522220 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:19.522848 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:19.522858 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:19.522889 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:42:19.523413 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:19.523425 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:19.525171 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:19.525176 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:19.527295 (+ 2119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:19.527307 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:19.529718 (+ 2411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:19.529725 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:19.529920 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:19.529961 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 16:42:19.530057 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 16:42:19.530185 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"mutex_wait_us":49,"thread_start_us":283,"threads_started":1}
I20260929 16:42:19.532883 8814 tablet_service.cc:1467] Processing CreateTablet for tablet e2396c6f3beb48ec9f7c2da8561f7176 (DEFAULT_TABLE table=test-table [id=8169cca393c647e8b1c79192db640765]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:42:19.533844 8669 tablet_service.cc:1467] Processing CreateTablet for tablet e2396c6f3beb48ec9f7c2da8561f7176 (DEFAULT_TABLE table=test-table [id=8169cca393c647e8b1c79192db640765]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:42:19.534463 8814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2396c6f3beb48ec9f7c2da8561f7176. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:42:19.534233 8529 tablet_service.cc:1467] Processing CreateTablet for tablet e2396c6f3beb48ec9f7c2da8561f7176 (DEFAULT_TABLE table=test-table [id=8169cca393c647e8b1c79192db640765]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:42:19.535246 8669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2396c6f3beb48ec9f7c2da8561f7176. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:42:19.535514 8529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2396c6f3beb48ec9f7c2da8561f7176. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:42:19.548632 8904 tablet_bootstrap.cc:492] T e2396c6f3beb48ec9f7c2da8561f7176 P 2e017281552241b99a107a7afd58b230: Bootstrap starting.
I20260929 16:42:19.552103 8904 tablet_bootstrap.cc:654] T e2396c6f3beb48ec9f7c2da8561f7176 P 2e017281552241b99a107a7afd58b230: Neither blocks nor log segments found. Creating new log.
I20260929 16:42:19.553049 8905 tablet_bootstrap.cc:492] T e2396c6f3beb48ec9f7c2da8561f7176 P ef398814e1ff47409b81c7fd8253ef15: Bootstrap starting.
I20260929 16:42:19.554591 8904 log.cc:824] T e2396c6f3beb48ec9f7c2da8561f7176 P 2e017281552241b99a107a7afd58b230: Log is configured to *not* fsync() on all Append() calls
I20260929 16:42:19.554769 8906 tablet_bootstrap.cc:492] T e2396c6f3beb48ec9f7c2da8561f7176 P a6d0ac696d0e42ad8808c989facfaee0: Bootstrap starting.
I20260929 16:42:19.558224 8906 tablet_bootstrap.cc:654] T e2396c6f3beb48ec9f7c2da8561f7176 P a6d0ac696d0e42ad8808c989facfaee0: Neither blocks nor log segments found. Creating new log.
I20260929 16:42:19.558679 8905 tablet_bootstrap.cc:654] T e2396c6f3beb48ec9f7c2da8561f7176 P ef398814e1ff47409b81c7fd8253ef15: Neither blocks nor log segments found. Creating new log.
I20260929 16:42:19.559736 8906 log.cc:824] T e2396c6f3beb48ec9f7c2da8561f7176 P a6d0ac696d0e42ad8808c989facfaee0: Log is configured to *not* fsync() on all Append() calls
I20260929 16:42:19.559960 8905 log.cc:824] T e2396c6f3beb48ec9f7c2da8561f7176 P ef398814e1ff47409b81c7fd8253ef15: Log is configured to *not* fsync() on all Append() calls
I20260929 16:42:19.561800 8904 tablet_bootstrap.cc:492] T e2396c6f3beb48ec9f7c2da8561f7176 P 2e017281552241b99a107a7afd58b230: No bootstrap required, opened a new log
I20260929 16:42:19.562129 8904 ts_tablet_manager.cc:1397] T e2396c6f3beb48ec9f7c2da8561f7176 P 2e017281552241b99a107a7afd58b230: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.000s
I20260929 16:42:19.563476 8905 tablet_bootstrap.cc:492] T e2396c6f3beb48ec9f7c2da8561f7176 P ef398814e1ff47409b81c7fd8253ef15: No bootstrap required, opened a new log
I20260929 16:42:19.563509 8906 tablet_bootstrap.cc:492] T e2396c6f3beb48ec9f7c2da8561f7176 P a6d0ac696d0e42ad8808c989facfaee0: No bootstrap required, opened a new log
I20260929 16:42:19.563802 8905 ts_tablet_manager.cc:1397] T e2396c6f3beb48ec9f7c2da8561f7176 P ef398814e1ff47409b81c7fd8253ef15: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260929 16:42:19.563807 8906 ts_tablet_manager.cc:1397] T e2396c6f3beb48ec9f7c2da8561f7176 P a6d0ac696d0e42ad8808c989facfaee0: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260929 16:42:19.572052 8905 raft_consensus.cc:348] T e2396c6f3beb48ec9f7c2da8561f7176 P ef398814e1ff47409b81c7fd8253ef15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } } peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } }
I20260929 16:42:19.572518 8905 raft_consensus.cc:374] T e2396c6f3beb48ec9f7c2da8561f7176 P ef398814e1ff47409b81c7fd8253ef15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:42:19.572639 8905 raft_consensus.cc:729] T e2396c6f3beb48ec9f7c2da8561f7176 P ef398814e1ff47409b81c7fd8253ef15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef398814e1ff47409b81c7fd8253ef15, State: Initialized, Role: FOLLOWER
I20260929 16:42:19.573328 8905 consensus_queue.cc:260] T e2396c6f3beb48ec9f7c2da8561f7176 P ef398814e1ff47409b81c7fd8253ef15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } } peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } }
I20260929 16:42:19.573541 8904 raft_consensus.cc:348] T e2396c6f3beb48ec9f7c2da8561f7176 P 2e017281552241b99a107a7afd58b230 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } } peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } }
I20260929 16:42:19.574085 8904 raft_consensus.cc:374] T e2396c6f3beb48ec9f7c2da8561f7176 P 2e017281552241b99a107a7afd58b230 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:42:19.575054 8905 ts_tablet_manager.cc:1428] T e2396c6f3beb48ec9f7c2da8561f7176 P ef398814e1ff47409b81c7fd8253ef15: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260929 16:42:19.574260 8904 raft_consensus.cc:729] T e2396c6f3beb48ec9f7c2da8561f7176 P 2e017281552241b99a107a7afd58b230 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e017281552241b99a107a7afd58b230, State: Initialized, Role: FOLLOWER
I20260929 16:42:19.575359 8906 raft_consensus.cc:348] T e2396c6f3beb48ec9f7c2da8561f7176 P a6d0ac696d0e42ad8808c989facfaee0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } } peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } }
I20260929 16:42:19.575850 8906 raft_consensus.cc:374] T e2396c6f3beb48ec9f7c2da8561f7176 P a6d0ac696d0e42ad8808c989facfaee0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:42:19.575763 8904 consensus_queue.cc:260] T e2396c6f3beb48ec9f7c2da8561f7176 P 2e017281552241b99a107a7afd58b230 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } } peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } }
I20260929 16:42:19.576001 8906 raft_consensus.cc:729] T e2396c6f3beb48ec9f7c2da8561f7176 P a6d0ac696d0e42ad8808c989facfaee0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6d0ac696d0e42ad8808c989facfaee0, State: Initialized, Role: FOLLOWER
I20260929 16:42:19.577654 8904 ts_tablet_manager.cc:1428] T e2396c6f3beb48ec9f7c2da8561f7176 P 2e017281552241b99a107a7afd58b230: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260929 16:42:19.580415 8906 consensus_queue.cc:260] T e2396c6f3beb48ec9f7c2da8561f7176 P a6d0ac696d0e42ad8808c989facfaee0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } } peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } }
I20260929 16:42:19.584071 8906 ts_tablet_manager.cc:1428] T e2396c6f3beb48ec9f7c2da8561f7176 P a6d0ac696d0e42ad8808c989facfaee0: Time spent starting tablet: real 0.020s user 0.015s sys 0.000s
W20260929 16:42:19.585047 8600 tablet.cc:2369] T e2396c6f3beb48ec9f7c2da8561f7176 P a6d0ac696d0e42ad8808c989facfaee0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:42:19.664763 8740 tablet.cc:2369] T e2396c6f3beb48ec9f7c2da8561f7176 P 2e017281552241b99a107a7afd58b230: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:42:19.701138 8911 raft_consensus.cc:482] T e2396c6f3beb48ec9f7c2da8561f7176 P 2e017281552241b99a107a7afd58b230 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:42:19.701424 8911 raft_consensus.cc:504] T e2396c6f3beb48ec9f7c2da8561f7176 P 2e017281552241b99a107a7afd58b230 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } } peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } }
I20260929 16:42:19.703586 8911 leader_election.cc:288] T e2396c6f3beb48ec9f7c2da8561f7176 P 2e017281552241b99a107a7afd58b230 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef398814e1ff47409b81c7fd8253ef15 (127.0.197.67:40929), a6d0ac696d0e42ad8808c989facfaee0 (127.0.197.65:44769)
I20260929 16:42:19.710670 8902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:19.704135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:60757 (local address 127.0.197.65:44769)
0929 16:42:19.704352 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:19.704358 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:19.705061 (+ 703us) server_negotiation.cc:407] Connection header received
0929 16:42:19.705260 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:19.705269 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:19.705327 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:19.705432 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:42:19.706078 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:19.706777 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:19.708682 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:19.709517 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:19.709713 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:19.709807 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:19.710268 (+ 461us) server_negotiation.cc:299] Negotiation successful
0929 16:42:19.710403 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 16:42:19.710680 8901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:19.703767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:49335 (local address 127.0.197.67:40929)
0929 16:42:19.704003 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:19.704011 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:19.704806 (+ 795us) server_negotiation.cc:407] Connection header received
0929 16:42:19.705070 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:19.705075 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:19.705134 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:19.705234 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:42:19.705796 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:19.706580 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:19.708719 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:19.709576 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:19.709762 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:19.709903 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:19.710245 (+ 342us) server_negotiation.cc:299] Negotiation successful
0929 16:42:19.710385 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 16:42:19.710841 8913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:19.703751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.67:40929 (local address 127.0.197.66:49335)
0929 16:42:19.704667 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:19.704683 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:19.704903 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:19.705249 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:19.705253 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:19.705286 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:42:19.705633 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:19.705642 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:19.706728 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:19.706733 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:19.708563 (+ 1830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:19.708575 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:19.709870 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:19.709878 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:19.710060 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:19.710103 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 16:42:19.710218 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 16:42:19.710385 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"mutex_wait_us":71,"thread_start_us":569,"threads_started":1}
I20260929 16:42:19.710932 8914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:19.704080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:44769 (local address 127.0.197.66:60757)
0929 16:42:19.704940 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:19.704958 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:19.705122 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:19.705445 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:19.705449 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:19.705476 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:42:19.705917 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:19.705927 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:19.706919 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:19.706924 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:19.708518 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:19.708533 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:19.709931 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:19.709942 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:19.710072 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:19.710122 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 16:42:19.710229 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 16:42:19.710389 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"mutex_wait_us":44,"thread_start_us":513,"threads_started":1}
I20260929 16:42:19.712157 8834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2396c6f3beb48ec9f7c2da8561f7176" candidate_uuid: "2e017281552241b99a107a7afd58b230" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef398814e1ff47409b81c7fd8253ef15" is_pre_election: true
I20260929 16:42:19.712304 8554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2396c6f3beb48ec9f7c2da8561f7176" candidate_uuid: "2e017281552241b99a107a7afd58b230" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6d0ac696d0e42ad8808c989facfaee0" is_pre_election: true
I20260929 16:42:19.712814 8834 raft_consensus.cc:2454] T e2396c6f3beb48ec9f7c2da8561f7176 P ef398814e1ff47409b81c7fd8253ef15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e017281552241b99a107a7afd58b230 in term 0.
I20260929 16:42:19.712814 8554 raft_consensus.cc:2454] T e2396c6f3beb48ec9f7c2da8561f7176 P a6d0ac696d0e42ad8808c989facfaee0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e017281552241b99a107a7afd58b230 in term 0.
I20260929 16:42:19.713608 8621 leader_election.cc:302] T e2396c6f3beb48ec9f7c2da8561f7176 P 2e017281552241b99a107a7afd58b230 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2e017281552241b99a107a7afd58b230, a6d0ac696d0e42ad8808c989facfaee0, ef398814e1ff47409b81c7fd8253ef15; no voters:
I20260929 16:42:19.714033 8911 raft_consensus.cc:2781] T e2396c6f3beb48ec9f7c2da8561f7176 P 2e017281552241b99a107a7afd58b230 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:42:19.714183 8911 raft_consensus.cc:482] T e2396c6f3beb48ec9f7c2da8561f7176 P 2e017281552241b99a107a7afd58b230 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:42:19.714294 8911 raft_consensus.cc:3037] T e2396c6f3beb48ec9f7c2da8561f7176 P 2e017281552241b99a107a7afd58b230 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:42:19.716552 8911 raft_consensus.cc:504] T e2396c6f3beb48ec9f7c2da8561f7176 P 2e017281552241b99a107a7afd58b230 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } } peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } }
I20260929 16:42:19.717214 8911 leader_election.cc:288] T e2396c6f3beb48ec9f7c2da8561f7176 P 2e017281552241b99a107a7afd58b230 [CANDIDATE]: Term 1 election: Requested vote from peers ef398814e1ff47409b81c7fd8253ef15 (127.0.197.67:40929), a6d0ac696d0e42ad8808c989facfaee0 (127.0.197.65:44769)
I20260929 16:42:19.717824 8554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2396c6f3beb48ec9f7c2da8561f7176" candidate_uuid: "2e017281552241b99a107a7afd58b230" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6d0ac696d0e42ad8808c989facfaee0"
I20260929 16:42:19.717829 8834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2396c6f3beb48ec9f7c2da8561f7176" candidate_uuid: "2e017281552241b99a107a7afd58b230" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef398814e1ff47409b81c7fd8253ef15"
I20260929 16:42:19.718065 8554 raft_consensus.cc:3037] T e2396c6f3beb48ec9f7c2da8561f7176 P a6d0ac696d0e42ad8808c989facfaee0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:42:19.718092 8834 raft_consensus.cc:3037] T e2396c6f3beb48ec9f7c2da8561f7176 P ef398814e1ff47409b81c7fd8253ef15 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:42:19.720425 8554 raft_consensus.cc:2454] T e2396c6f3beb48ec9f7c2da8561f7176 P a6d0ac696d0e42ad8808c989facfaee0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e017281552241b99a107a7afd58b230 in term 1.
I20260929 16:42:19.720425 8834 raft_consensus.cc:2454] T e2396c6f3beb48ec9f7c2da8561f7176 P ef398814e1ff47409b81c7fd8253ef15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e017281552241b99a107a7afd58b230 in term 1.
I20260929 16:42:19.721138 8621 leader_election.cc:302] T e2396c6f3beb48ec9f7c2da8561f7176 P 2e017281552241b99a107a7afd58b230 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e017281552241b99a107a7afd58b230, a6d0ac696d0e42ad8808c989facfaee0; no voters:
I20260929 16:42:19.721516 8911 raft_consensus.cc:2781] T e2396c6f3beb48ec9f7c2da8561f7176 P 2e017281552241b99a107a7afd58b230 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:42:19.722083 8911 raft_consensus.cc:686] T e2396c6f3beb48ec9f7c2da8561f7176 P 2e017281552241b99a107a7afd58b230 [term 1 LEADER]: Becoming Leader. State: Replica: 2e017281552241b99a107a7afd58b230, State: Running, Role: LEADER
I20260929 16:42:19.722630 8911 consensus_queue.cc:237] T e2396c6f3beb48ec9f7c2da8561f7176 P 2e017281552241b99a107a7afd58b230 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } } peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } }
I20260929 16:42:19.727350 8391 catalog_manager.cc:5495] T e2396c6f3beb48ec9f7c2da8561f7176 P 2e017281552241b99a107a7afd58b230 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e017281552241b99a107a7afd58b230 (127.0.197.66). New cstate: current_term: 1 leader_uuid: "2e017281552241b99a107a7afd58b230" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } health_report { overall_health: UNKNOWN } } }
I20260929 16:42:19.782358 8899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:19.776160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58778 (local address 127.0.197.66:42265)
0929 16:42:19.776380 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:19.776385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:19.776404 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:42:19.776610 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:19.776614 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:19.776697 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:19.776860 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:42:19.777158 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:19.777884 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:19.779450 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:19.779922 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:19.780048 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:19.780532 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:42:19.781136 (+ 604us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:42:19.781280 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:19.781607 (+ 327us) server_negotiation.cc:299] Negotiation successful
0929 16:42:19.781777 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 16:42:19.791546 8554 raft_consensus.cc:1261] T e2396c6f3beb48ec9f7c2da8561f7176 P a6d0ac696d0e42ad8808c989facfaee0 [term 1 FOLLOWER]: Refusing update from remote peer 2e017281552241b99a107a7afd58b230: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:42:19.791652 8834 raft_consensus.cc:1261] T e2396c6f3beb48ec9f7c2da8561f7176 P ef398814e1ff47409b81c7fd8253ef15 [term 1 FOLLOWER]: Refusing update from remote peer 2e017281552241b99a107a7afd58b230: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:42:19.792747 8915 consensus_queue.cc:1035] T e2396c6f3beb48ec9f7c2da8561f7176 P 2e017281552241b99a107a7afd58b230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:42:19.793169 8911 consensus_queue.cc:1035] T e2396c6f3beb48ec9f7c2da8561f7176 P 2e017281552241b99a107a7afd58b230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 16:42:19.805096 8880 tablet.cc:2369] T e2396c6f3beb48ec9f7c2da8561f7176 P ef398814e1ff47409b81c7fd8253ef15: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:42:19.808663 8918 mvcc.cc:205] Tried to move back new op lower bound from 7334707772575473664 to 7334707772308000768. Current Snapshot: MvccSnapshot[applied={T|T < 7334707772575473664}]
I20260929 16:42:19.824193 8391 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:42:19.824528 8391 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:42:19.830305 8391 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a62329d5a3f14fc2846e3e1a948f8aa9: Sending DeleteTablet for 3 replicas of tablet e2396c6f3beb48ec9f7c2da8561f7176
I20260929 16:42:19.832006 8669 tablet_service.cc:1514] Processing DeleteTablet for tablet e2396c6f3beb48ec9f7c2da8561f7176 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:42:19 UTC) from {username='oryx', principal='oryx/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:58764
I20260929 16:42:19.832643 8529 tablet_service.cc:1514] Processing DeleteTablet for tablet e2396c6f3beb48ec9f7c2da8561f7176 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:42:19 UTC) from {username='oryx', principal='oryx/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:51312
I20260929 16:42:19.832737 8528 tablet_service.cc:1514] Processing DeleteTablet for tablet e2396c6f3beb48ec9f7c2da8561f7176 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:42:19 UTC) from {username='oryx', principal='oryx/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:51312
I20260929 16:42:19.833237 8814 tablet_service.cc:1514] Processing DeleteTablet for tablet e2396c6f3beb48ec9f7c2da8561f7176 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:42:19 UTC) from {username='oryx', principal='oryx/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:53382
I20260929 16:42:19.833362 8813 tablet_service.cc:1514] Processing DeleteTablet for tablet e2396c6f3beb48ec9f7c2da8561f7176 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:42:19 UTC) from {username='oryx', principal='oryx/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:53382
I20260929 16:42:19.833482 8391 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54176:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:42:19.834374 8391 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:42:19.838083 8929 tablet_replica.cc:330] T e2396c6f3beb48ec9f7c2da8561f7176 P ef398814e1ff47409b81c7fd8253ef15: stopping tablet replica
I20260929 16:42:19.838709 8929 raft_consensus.cc:2229] T e2396c6f3beb48ec9f7c2da8561f7176 P ef398814e1ff47409b81c7fd8253ef15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:42:19.840701 8929 raft_consensus.cc:2258] T e2396c6f3beb48ec9f7c2da8561f7176 P ef398814e1ff47409b81c7fd8253ef15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:42:19.843623 8929 ts_tablet_manager.cc:1905] T e2396c6f3beb48ec9f7c2da8561f7176 P ef398814e1ff47409b81c7fd8253ef15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:42:19.844466 8928 tablet_replica.cc:330] T e2396c6f3beb48ec9f7c2da8561f7176 P a6d0ac696d0e42ad8808c989facfaee0: stopping tablet replica
I20260929 16:42:19.845028 8928 raft_consensus.cc:2229] T e2396c6f3beb48ec9f7c2da8561f7176 P a6d0ac696d0e42ad8808c989facfaee0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:42:19.845198 8814 tablet_service.cc:1467] Processing CreateTablet for tablet 25a691dde75740d0aa9167e6ff021195 (DEFAULT_TABLE table=test-table [id=c6d9f02fc7224144bf53b1ae17b26cd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:42:19.845677 8814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25a691dde75740d0aa9167e6ff021195. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:42:19.845920 8529 tablet_service.cc:1467] Processing CreateTablet for tablet 25a691dde75740d0aa9167e6ff021195 (DEFAULT_TABLE table=test-table [id=c6d9f02fc7224144bf53b1ae17b26cd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:42:19.846072 8928 raft_consensus.cc:2258] T e2396c6f3beb48ec9f7c2da8561f7176 P a6d0ac696d0e42ad8808c989facfaee0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:42:19.846360 8529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25a691dde75740d0aa9167e6ff021195. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:42:19.847609 8668 tablet_service.cc:1467] Processing CreateTablet for tablet 25a691dde75740d0aa9167e6ff021195 (DEFAULT_TABLE table=test-table [id=c6d9f02fc7224144bf53b1ae17b26cd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:42:19.848408 8668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25a691dde75740d0aa9167e6ff021195. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:42:19.854130 8929 ts_tablet_manager.cc:1918] T e2396c6f3beb48ec9f7c2da8561f7176 P ef398814e1ff47409b81c7fd8253ef15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:42:19.854456 8929 log.cc:1196] T e2396c6f3beb48ec9f7c2da8561f7176 P ef398814e1ff47409b81c7fd8253ef15: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-2/wal/wals/e2396c6f3beb48ec9f7c2da8561f7176
I20260929 16:42:19.855228 8929 ts_tablet_manager.cc:1939] T e2396c6f3beb48ec9f7c2da8561f7176 P ef398814e1ff47409b81c7fd8253ef15: Deleting consensus metadata
I20260929 16:42:19.856599 8927 tablet_replica.cc:330] T e2396c6f3beb48ec9f7c2da8561f7176 P 2e017281552241b99a107a7afd58b230: stopping tablet replica
I20260929 16:42:19.857281 8927 raft_consensus.cc:2229] T e2396c6f3beb48ec9f7c2da8561f7176 P 2e017281552241b99a107a7afd58b230 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:42:19.858127 8927 raft_consensus.cc:2258] T e2396c6f3beb48ec9f7c2da8561f7176 P 2e017281552241b99a107a7afd58b230 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:42:19.858788 8906 tablet_bootstrap.cc:492] T 25a691dde75740d0aa9167e6ff021195 P a6d0ac696d0e42ad8808c989facfaee0: Bootstrap starting.
I20260929 16:42:19.860546 8376 catalog_manager.cc:4841] TS ef398814e1ff47409b81c7fd8253ef15 (127.0.197.67:40929): tablet e2396c6f3beb48ec9f7c2da8561f7176 (table test-table [id=8169cca393c647e8b1c79192db640765]) successfully deleted
I20260929 16:42:19.860623 8927 ts_tablet_manager.cc:1905] T e2396c6f3beb48ec9f7c2da8561f7176 P 2e017281552241b99a107a7afd58b230: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:42:19.860685 8906 tablet_bootstrap.cc:654] T 25a691dde75740d0aa9167e6ff021195 P a6d0ac696d0e42ad8808c989facfaee0: Neither blocks nor log segments found. Creating new log.
W20260929 16:42:19.861066 8376 catalog_manager.cc:4808] TS ef398814e1ff47409b81c7fd8253ef15 (127.0.197.67:40929): delete failed for tablet e2396c6f3beb48ec9f7c2da8561f7176 because the tablet was not found. No further retry: Not found: Tablet not found: e2396c6f3beb48ec9f7c2da8561f7176
I20260929 16:42:19.867110 8928 ts_tablet_manager.cc:1905] T e2396c6f3beb48ec9f7c2da8561f7176 P a6d0ac696d0e42ad8808c989facfaee0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:42:19.869122 8904 tablet_bootstrap.cc:492] T 25a691dde75740d0aa9167e6ff021195 P 2e017281552241b99a107a7afd58b230: Bootstrap starting.
I20260929 16:42:19.871098 8904 tablet_bootstrap.cc:654] T 25a691dde75740d0aa9167e6ff021195 P 2e017281552241b99a107a7afd58b230: Neither blocks nor log segments found. Creating new log.
I20260929 16:42:19.873683 8927 ts_tablet_manager.cc:1918] T e2396c6f3beb48ec9f7c2da8561f7176 P 2e017281552241b99a107a7afd58b230: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:42:19.874019 8927 log.cc:1196] T e2396c6f3beb48ec9f7c2da8561f7176 P 2e017281552241b99a107a7afd58b230: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-1/wal/wals/e2396c6f3beb48ec9f7c2da8561f7176
I20260929 16:42:19.874816 8927 ts_tablet_manager.cc:1939] T e2396c6f3beb48ec9f7c2da8561f7176 P 2e017281552241b99a107a7afd58b230: Deleting consensus metadata
I20260929 16:42:19.877153 8378 catalog_manager.cc:4841] TS 2e017281552241b99a107a7afd58b230 (127.0.197.66:42265): tablet e2396c6f3beb48ec9f7c2da8561f7176 (table test-table [id=8169cca393c647e8b1c79192db640765]) successfully deleted
I20260929 16:42:19.880720 8904 tablet_bootstrap.cc:492] T 25a691dde75740d0aa9167e6ff021195 P 2e017281552241b99a107a7afd58b230: No bootstrap required, opened a new log
I20260929 16:42:19.881228 8904 ts_tablet_manager.cc:1397] T 25a691dde75740d0aa9167e6ff021195 P 2e017281552241b99a107a7afd58b230: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.006s
I20260929 16:42:19.882287 8904 raft_consensus.cc:348] T 25a691dde75740d0aa9167e6ff021195 P 2e017281552241b99a107a7afd58b230 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } } peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } }
I20260929 16:42:19.882669 8904 raft_consensus.cc:374] T 25a691dde75740d0aa9167e6ff021195 P 2e017281552241b99a107a7afd58b230 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:42:19.882828 8904 raft_consensus.cc:729] T 25a691dde75740d0aa9167e6ff021195 P 2e017281552241b99a107a7afd58b230 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e017281552241b99a107a7afd58b230, State: Initialized, Role: FOLLOWER
I20260929 16:42:19.883170 8904 consensus_queue.cc:260] T 25a691dde75740d0aa9167e6ff021195 P 2e017281552241b99a107a7afd58b230 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } } peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } }
I20260929 16:42:19.883868 8904 ts_tablet_manager.cc:1428] T 25a691dde75740d0aa9167e6ff021195 P 2e017281552241b99a107a7afd58b230: Time spent starting tablet: real 0.002s user 0.001s sys 0.003s
I20260929 16:42:19.884567 8905 tablet_bootstrap.cc:492] T 25a691dde75740d0aa9167e6ff021195 P ef398814e1ff47409b81c7fd8253ef15: Bootstrap starting.
I20260929 16:42:19.886348 8906 tablet_bootstrap.cc:492] T 25a691dde75740d0aa9167e6ff021195 P a6d0ac696d0e42ad8808c989facfaee0: No bootstrap required, opened a new log
I20260929 16:42:19.886507 8905 tablet_bootstrap.cc:654] T 25a691dde75740d0aa9167e6ff021195 P ef398814e1ff47409b81c7fd8253ef15: Neither blocks nor log segments found. Creating new log.
I20260929 16:42:19.886530 8906 ts_tablet_manager.cc:1397] T 25a691dde75740d0aa9167e6ff021195 P a6d0ac696d0e42ad8808c989facfaee0: Time spent bootstrapping tablet: real 0.028s user 0.008s sys 0.004s
I20260929 16:42:19.887346 8906 raft_consensus.cc:348] T 25a691dde75740d0aa9167e6ff021195 P a6d0ac696d0e42ad8808c989facfaee0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } } peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } }
I20260929 16:42:19.887563 8906 raft_consensus.cc:374] T 25a691dde75740d0aa9167e6ff021195 P a6d0ac696d0e42ad8808c989facfaee0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:42:19.887661 8906 raft_consensus.cc:729] T 25a691dde75740d0aa9167e6ff021195 P a6d0ac696d0e42ad8808c989facfaee0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6d0ac696d0e42ad8808c989facfaee0, State: Initialized, Role: FOLLOWER
I20260929 16:42:19.887946 8906 consensus_queue.cc:260] T 25a691dde75740d0aa9167e6ff021195 P a6d0ac696d0e42ad8808c989facfaee0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } } peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } }
I20260929 16:42:19.888500 8906 ts_tablet_manager.cc:1428] T 25a691dde75740d0aa9167e6ff021195 P a6d0ac696d0e42ad8808c989facfaee0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 16:42:19.891142 8905 tablet_bootstrap.cc:492] T 25a691dde75740d0aa9167e6ff021195 P ef398814e1ff47409b81c7fd8253ef15: No bootstrap required, opened a new log
I20260929 16:42:19.891450 8905 ts_tablet_manager.cc:1397] T 25a691dde75740d0aa9167e6ff021195 P ef398814e1ff47409b81c7fd8253ef15: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 16:42:19.892473 8905 raft_consensus.cc:348] T 25a691dde75740d0aa9167e6ff021195 P ef398814e1ff47409b81c7fd8253ef15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } } peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } }
I20260929 16:42:19.892845 8905 raft_consensus.cc:374] T 25a691dde75740d0aa9167e6ff021195 P ef398814e1ff47409b81c7fd8253ef15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:42:19.893007 8905 raft_consensus.cc:729] T 25a691dde75740d0aa9167e6ff021195 P ef398814e1ff47409b81c7fd8253ef15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef398814e1ff47409b81c7fd8253ef15, State: Initialized, Role: FOLLOWER
I20260929 16:42:19.893364 8905 consensus_queue.cc:260] T 25a691dde75740d0aa9167e6ff021195 P ef398814e1ff47409b81c7fd8253ef15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } } peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } }
I20260929 16:42:19.893530 8928 ts_tablet_manager.cc:1918] T e2396c6f3beb48ec9f7c2da8561f7176 P a6d0ac696d0e42ad8808c989facfaee0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:42:19.893810 8928 log.cc:1196] T e2396c6f3beb48ec9f7c2da8561f7176 P a6d0ac696d0e42ad8808c989facfaee0: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-0/wal/wals/e2396c6f3beb48ec9f7c2da8561f7176
I20260929 16:42:19.894508 8928 ts_tablet_manager.cc:1939] T e2396c6f3beb48ec9f7c2da8561f7176 P a6d0ac696d0e42ad8808c989facfaee0: Deleting consensus metadata
I20260929 16:42:19.895475 8905 ts_tablet_manager.cc:1428] T 25a691dde75740d0aa9167e6ff021195 P ef398814e1ff47409b81c7fd8253ef15: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260929 16:42:19.896893 8378 catalog_manager.cc:4841] TS a6d0ac696d0e42ad8808c989facfaee0 (127.0.197.65:44769): tablet e2396c6f3beb48ec9f7c2da8561f7176 (table test-table [id=8169cca393c647e8b1c79192db640765]) successfully deleted
W20260929 16:42:19.897342 8378 catalog_manager.cc:4808] TS a6d0ac696d0e42ad8808c989facfaee0 (127.0.197.65:44769): delete failed for tablet e2396c6f3beb48ec9f7c2da8561f7176 because the tablet was not found. No further retry: Not found: Tablet not found: e2396c6f3beb48ec9f7c2da8561f7176
I20260929 16:42:19.961336 8911 raft_consensus.cc:482] T 25a691dde75740d0aa9167e6ff021195 P 2e017281552241b99a107a7afd58b230 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:42:19.961560 8911 raft_consensus.cc:504] T 25a691dde75740d0aa9167e6ff021195 P 2e017281552241b99a107a7afd58b230 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } } peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } }
I20260929 16:42:19.962337 8911 leader_election.cc:288] T 25a691dde75740d0aa9167e6ff021195 P 2e017281552241b99a107a7afd58b230 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6d0ac696d0e42ad8808c989facfaee0 (127.0.197.65:44769), ef398814e1ff47409b81c7fd8253ef15 (127.0.197.67:40929)
I20260929 16:42:19.962983 8834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25a691dde75740d0aa9167e6ff021195" candidate_uuid: "2e017281552241b99a107a7afd58b230" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef398814e1ff47409b81c7fd8253ef15" is_pre_election: true
I20260929 16:42:19.963081 8553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25a691dde75740d0aa9167e6ff021195" candidate_uuid: "2e017281552241b99a107a7afd58b230" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6d0ac696d0e42ad8808c989facfaee0" is_pre_election: true
I20260929 16:42:19.963286 8834 raft_consensus.cc:2454] T 25a691dde75740d0aa9167e6ff021195 P ef398814e1ff47409b81c7fd8253ef15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e017281552241b99a107a7afd58b230 in term 0.
I20260929 16:42:19.963455 8553 raft_consensus.cc:2454] T 25a691dde75740d0aa9167e6ff021195 P a6d0ac696d0e42ad8808c989facfaee0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e017281552241b99a107a7afd58b230 in term 0.
I20260929 16:42:19.963750 8619 leader_election.cc:302] T 25a691dde75740d0aa9167e6ff021195 P 2e017281552241b99a107a7afd58b230 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e017281552241b99a107a7afd58b230, ef398814e1ff47409b81c7fd8253ef15; no voters:
I20260929 16:42:19.964072 8911 raft_consensus.cc:2781] T 25a691dde75740d0aa9167e6ff021195 P 2e017281552241b99a107a7afd58b230 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:42:19.964248 8911 raft_consensus.cc:482] T 25a691dde75740d0aa9167e6ff021195 P 2e017281552241b99a107a7afd58b230 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:42:19.964397 8911 raft_consensus.cc:3037] T 25a691dde75740d0aa9167e6ff021195 P 2e017281552241b99a107a7afd58b230 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:42:19.967247 8911 raft_consensus.cc:504] T 25a691dde75740d0aa9167e6ff021195 P 2e017281552241b99a107a7afd58b230 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } } peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } }
I20260929 16:42:19.967980 8911 leader_election.cc:288] T 25a691dde75740d0aa9167e6ff021195 P 2e017281552241b99a107a7afd58b230 [CANDIDATE]: Term 1 election: Requested vote from peers a6d0ac696d0e42ad8808c989facfaee0 (127.0.197.65:44769), ef398814e1ff47409b81c7fd8253ef15 (127.0.197.67:40929)
I20260929 16:42:19.968541 8553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25a691dde75740d0aa9167e6ff021195" candidate_uuid: "2e017281552241b99a107a7afd58b230" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6d0ac696d0e42ad8808c989facfaee0"
I20260929 16:42:19.968597 8834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25a691dde75740d0aa9167e6ff021195" candidate_uuid: "2e017281552241b99a107a7afd58b230" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef398814e1ff47409b81c7fd8253ef15"
I20260929 16:42:19.968812 8553 raft_consensus.cc:3037] T 25a691dde75740d0aa9167e6ff021195 P a6d0ac696d0e42ad8808c989facfaee0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:42:19.968832 8834 raft_consensus.cc:3037] T 25a691dde75740d0aa9167e6ff021195 P ef398814e1ff47409b81c7fd8253ef15 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:42:19.972239 8553 raft_consensus.cc:2454] T 25a691dde75740d0aa9167e6ff021195 P a6d0ac696d0e42ad8808c989facfaee0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e017281552241b99a107a7afd58b230 in term 1.
I20260929 16:42:19.972239 8834 raft_consensus.cc:2454] T 25a691dde75740d0aa9167e6ff021195 P ef398814e1ff47409b81c7fd8253ef15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e017281552241b99a107a7afd58b230 in term 1.
I20260929 16:42:19.972813 8619 leader_election.cc:302] T 25a691dde75740d0aa9167e6ff021195 P 2e017281552241b99a107a7afd58b230 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e017281552241b99a107a7afd58b230, ef398814e1ff47409b81c7fd8253ef15; no voters:
I20260929 16:42:19.973241 8911 raft_consensus.cc:2781] T 25a691dde75740d0aa9167e6ff021195 P 2e017281552241b99a107a7afd58b230 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:42:19.973416 8911 raft_consensus.cc:686] T 25a691dde75740d0aa9167e6ff021195 P 2e017281552241b99a107a7afd58b230 [term 1 LEADER]: Becoming Leader. State: Replica: 2e017281552241b99a107a7afd58b230, State: Running, Role: LEADER
I20260929 16:42:19.973759 8911 consensus_queue.cc:237] T 25a691dde75740d0aa9167e6ff021195 P 2e017281552241b99a107a7afd58b230 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } } peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } }
I20260929 16:42:19.976953 8392 catalog_manager.cc:5495] T 25a691dde75740d0aa9167e6ff021195 P 2e017281552241b99a107a7afd58b230 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e017281552241b99a107a7afd58b230 (127.0.197.66). New cstate: current_term: 1 leader_uuid: "2e017281552241b99a107a7afd58b230" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } health_report { overall_health: UNKNOWN } } }
Sep 29 16:42:20 dist-test-slave-l31f krb5kdc[8347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700136, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.197.126@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM
I20260929 16:42:20.030769 8939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:20.014142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:38787 (local address 127.0.0.1:54190)
0929 16:42:20.015343 (+ 1201us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:20.015373 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:20.015524 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:20.015925 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:20.015933 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:20.016695 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:20.016977 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:20.016987 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:20.018619 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:20.018623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:20.019659 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:20.019679 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:20.020675 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:20.020680 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:20.020824 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:20.021626 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:20.021644 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:20.024015 (+ 2371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:42:20.028189 (+ 4174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:20.028202 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:20.028209 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:20.028673 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:20.029101 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:20.029105 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:20.029108 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:20.029282 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:20.029934 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:42:20.029941 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:42:20.030102 (+ 161us) client_negotiation.cc:769] Sending connection context
0929 16:42:20.030258 (+ 156us) client_negotiation.cc:240] Negotiation successful
0929 16:42:20.030498 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1031,"thread_start_us":795,"threads_started":1}
I20260929 16:42:20.031217 8940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:20.014144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54190 (local address 127.0.197.126:38787)
0929 16:42:20.015073 (+ 929us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:20.015082 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:20.015484 (+ 402us) server_negotiation.cc:407] Connection header received
0929 16:42:20.015646 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:20.015652 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:20.015738 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:20.015862 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:20.017174 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:20.018460 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:20.019822 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:20.020488 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:20.020626 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:20.024281 (+ 3655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:20.024288 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:20.024293 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:20.024332 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:20.027922 (+ 3590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:20.028824 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:20.028830 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:20.028835 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:20.028990 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:20.029411 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:20.029417 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:20.029420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:20.029814 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:20.029961 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:20.030374 (+ 413us) server_negotiation.cc:299] Negotiation successful
0929 16:42:20.030695 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":688,"thread_start_us":446,"threads_started":1}
I20260929 16:42:20.037055 8392 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:54190:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 16:42:20.038177 8392 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:42:20.046007 8668 tablet_service.cc:1467] Processing CreateTablet for tablet 3b8e4b10e4794f71a7cf01bdb13eefef (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=129722537f7743bc88b7d2f68b63e055]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 16:42:20.045977 8529 tablet_service.cc:1467] Processing CreateTablet for tablet 3b8e4b10e4794f71a7cf01bdb13eefef (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=129722537f7743bc88b7d2f68b63e055]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 16:42:20.046212 8814 tablet_service.cc:1467] Processing CreateTablet for tablet 3b8e4b10e4794f71a7cf01bdb13eefef (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=129722537f7743bc88b7d2f68b63e055]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 16:42:20.046583 8668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b8e4b10e4794f71a7cf01bdb13eefef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:42:20.046583 8529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b8e4b10e4794f71a7cf01bdb13eefef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:42:20.046813 8814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b8e4b10e4794f71a7cf01bdb13eefef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:42:20.054548 8906 tablet_bootstrap.cc:492] T 3b8e4b10e4794f71a7cf01bdb13eefef P a6d0ac696d0e42ad8808c989facfaee0: Bootstrap starting.
I20260929 16:42:20.054594 8904 tablet_bootstrap.cc:492] T 3b8e4b10e4794f71a7cf01bdb13eefef P 2e017281552241b99a107a7afd58b230: Bootstrap starting.
I20260929 16:42:20.056414 8906 tablet_bootstrap.cc:654] T 3b8e4b10e4794f71a7cf01bdb13eefef P a6d0ac696d0e42ad8808c989facfaee0: Neither blocks nor log segments found. Creating new log.
I20260929 16:42:20.056470 8904 tablet_bootstrap.cc:654] T 3b8e4b10e4794f71a7cf01bdb13eefef P 2e017281552241b99a107a7afd58b230: Neither blocks nor log segments found. Creating new log.
I20260929 16:42:20.058527 8906 tablet_bootstrap.cc:492] T 3b8e4b10e4794f71a7cf01bdb13eefef P a6d0ac696d0e42ad8808c989facfaee0: No bootstrap required, opened a new log
I20260929 16:42:20.058746 8906 ts_tablet_manager.cc:1397] T 3b8e4b10e4794f71a7cf01bdb13eefef P a6d0ac696d0e42ad8808c989facfaee0: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 16:42:20.059599 8906 raft_consensus.cc:348] T 3b8e4b10e4794f71a7cf01bdb13eefef P a6d0ac696d0e42ad8808c989facfaee0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } } peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } }
I20260929 16:42:20.059851 8906 raft_consensus.cc:374] T 3b8e4b10e4794f71a7cf01bdb13eefef P a6d0ac696d0e42ad8808c989facfaee0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:42:20.059929 8906 raft_consensus.cc:729] T 3b8e4b10e4794f71a7cf01bdb13eefef P a6d0ac696d0e42ad8808c989facfaee0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6d0ac696d0e42ad8808c989facfaee0, State: Initialized, Role: FOLLOWER
I20260929 16:42:20.060168 8906 consensus_queue.cc:260] T 3b8e4b10e4794f71a7cf01bdb13eefef P a6d0ac696d0e42ad8808c989facfaee0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } } peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } }
I20260929 16:42:20.060353 8905 tablet_bootstrap.cc:492] T 3b8e4b10e4794f71a7cf01bdb13eefef P ef398814e1ff47409b81c7fd8253ef15: Bootstrap starting.
I20260929 16:42:20.060799 8906 ts_tablet_manager.cc:1428] T 3b8e4b10e4794f71a7cf01bdb13eefef P a6d0ac696d0e42ad8808c989facfaee0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 16:42:20.060642 8912 tablet_replica.cc:439] T 3b8e4b10e4794f71a7cf01bdb13eefef P a6d0ac696d0e42ad8808c989facfaee0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } } peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } } }
I20260929 16:42:20.061034 8912 tablet_replica.cc:442] T 3b8e4b10e4794f71a7cf01bdb13eefef P a6d0ac696d0e42ad8808c989facfaee0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:42:20.062131 8905 tablet_bootstrap.cc:654] T 3b8e4b10e4794f71a7cf01bdb13eefef P ef398814e1ff47409b81c7fd8253ef15: Neither blocks nor log segments found. Creating new log.
I20260929 16:42:20.065325 8905 tablet_bootstrap.cc:492] T 3b8e4b10e4794f71a7cf01bdb13eefef P ef398814e1ff47409b81c7fd8253ef15: No bootstrap required, opened a new log
I20260929 16:42:20.065327 8904 tablet_bootstrap.cc:492] T 3b8e4b10e4794f71a7cf01bdb13eefef P 2e017281552241b99a107a7afd58b230: No bootstrap required, opened a new log
I20260929 16:42:20.065500 8905 ts_tablet_manager.cc:1397] T 3b8e4b10e4794f71a7cf01bdb13eefef P ef398814e1ff47409b81c7fd8253ef15: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260929 16:42:20.065528 8904 ts_tablet_manager.cc:1397] T 3b8e4b10e4794f71a7cf01bdb13eefef P 2e017281552241b99a107a7afd58b230: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.004s
I20260929 16:42:20.066382 8905 raft_consensus.cc:348] T 3b8e4b10e4794f71a7cf01bdb13eefef P ef398814e1ff47409b81c7fd8253ef15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } } peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } }
I20260929 16:42:20.066395 8904 raft_consensus.cc:348] T 3b8e4b10e4794f71a7cf01bdb13eefef P 2e017281552241b99a107a7afd58b230 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } } peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } }
I20260929 16:42:20.066666 8904 raft_consensus.cc:374] T 3b8e4b10e4794f71a7cf01bdb13eefef P 2e017281552241b99a107a7afd58b230 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:42:20.066777 8904 raft_consensus.cc:729] T 3b8e4b10e4794f71a7cf01bdb13eefef P 2e017281552241b99a107a7afd58b230 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e017281552241b99a107a7afd58b230, State: Initialized, Role: FOLLOWER
I20260929 16:42:20.066799 8905 raft_consensus.cc:374] T 3b8e4b10e4794f71a7cf01bdb13eefef P ef398814e1ff47409b81c7fd8253ef15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:42:20.067051 8905 raft_consensus.cc:729] T 3b8e4b10e4794f71a7cf01bdb13eefef P ef398814e1ff47409b81c7fd8253ef15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef398814e1ff47409b81c7fd8253ef15, State: Initialized, Role: FOLLOWER
I20260929 16:42:20.067147 8904 consensus_queue.cc:260] T 3b8e4b10e4794f71a7cf01bdb13eefef P 2e017281552241b99a107a7afd58b230 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } } peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } }
I20260929 16:42:20.067373 8905 consensus_queue.cc:260] T 3b8e4b10e4794f71a7cf01bdb13eefef P ef398814e1ff47409b81c7fd8253ef15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } } peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } }
I20260929 16:42:20.067619 8911 tablet_replica.cc:439] T 3b8e4b10e4794f71a7cf01bdb13eefef P 2e017281552241b99a107a7afd58b230: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } } peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } } }
I20260929 16:42:20.067956 8911 tablet_replica.cc:442] T 3b8e4b10e4794f71a7cf01bdb13eefef P 2e017281552241b99a107a7afd58b230: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:42:20.067977 8905 ts_tablet_manager.cc:1428] T 3b8e4b10e4794f71a7cf01bdb13eefef P ef398814e1ff47409b81c7fd8253ef15: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 16:42:20.068485 8904 ts_tablet_manager.cc:1428] T 3b8e4b10e4794f71a7cf01bdb13eefef P 2e017281552241b99a107a7afd58b230: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 16:42:20.068306 8910 tablet_replica.cc:439] T 3b8e4b10e4794f71a7cf01bdb13eefef P ef398814e1ff47409b81c7fd8253ef15: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } } peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } } }
I20260929 16:42:20.068661 8910 tablet_replica.cc:442] T 3b8e4b10e4794f71a7cf01bdb13eefef P ef398814e1ff47409b81c7fd8253ef15: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:42:20.242132 8910 raft_consensus.cc:482] T 3b8e4b10e4794f71a7cf01bdb13eefef P ef398814e1ff47409b81c7fd8253ef15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:42:20.242352 8910 raft_consensus.cc:504] T 3b8e4b10e4794f71a7cf01bdb13eefef P ef398814e1ff47409b81c7fd8253ef15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } } peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } }
I20260929 16:42:20.244038 8910 leader_election.cc:288] T 3b8e4b10e4794f71a7cf01bdb13eefef P ef398814e1ff47409b81c7fd8253ef15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6d0ac696d0e42ad8808c989facfaee0 (127.0.197.65:44769), 2e017281552241b99a107a7afd58b230 (127.0.197.66:42265)
I20260929 16:42:20.245926 8912 raft_consensus.cc:482] T 3b8e4b10e4794f71a7cf01bdb13eefef P a6d0ac696d0e42ad8808c989facfaee0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:42:20.246150 8912 raft_consensus.cc:504] T 3b8e4b10e4794f71a7cf01bdb13eefef P a6d0ac696d0e42ad8808c989facfaee0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } } peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } }
I20260929 16:42:20.248356 8912 leader_election.cc:288] T 3b8e4b10e4794f71a7cf01bdb13eefef P a6d0ac696d0e42ad8808c989facfaee0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e017281552241b99a107a7afd58b230 (127.0.197.66:42265), ef398814e1ff47409b81c7fd8253ef15 (127.0.197.67:40929)
I20260929 16:42:20.270203 8899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:20.245565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:45057 (local address 127.0.197.66:42265)
0929 16:42:20.245782 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:20.245787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:20.246352 (+ 565us) server_negotiation.cc:407] Connection header received
0929 16:42:20.246619 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:20.246626 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:20.246700 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:20.246763 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:42:20.247682 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:20.248856 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:20.264966 (+ 16110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:20.266878 (+ 1912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:20.267115 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:20.267236 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:20.269495 (+ 2259us) server_negotiation.cc:299] Negotiation successful
0929 16:42:20.269737 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260929 16:42:20.271876 8946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:20.245167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.66:42265 (local address 127.0.197.67:45057)
0929 16:42:20.246244 (+ 1077us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:20.246264 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:20.246457 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:20.246995 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:20.246999 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:20.247066 (+ 67us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:42:20.247460 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:20.247472 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:20.249095 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:20.249101 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:20.264116 (+ 15015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:20.264150 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:20.267463 (+ 3313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:20.267473 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:20.267764 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:20.267820 (+ 56us) client_negotiation.cc:769] Sending connection context
0929 16:42:20.271255 (+ 3435us) client_negotiation.cc:240] Negotiation successful
0929 16:42:20.271463 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":951,"thread_start_us":775,"threads_started":1}
I20260929 16:42:20.272871 8689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b8e4b10e4794f71a7cf01bdb13eefef" candidate_uuid: "ef398814e1ff47409b81c7fd8253ef15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e017281552241b99a107a7afd58b230" is_pre_election: true
I20260929 16:42:20.273337 8689 raft_consensus.cc:2454] T 3b8e4b10e4794f71a7cf01bdb13eefef P 2e017281552241b99a107a7afd58b230 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef398814e1ff47409b81c7fd8253ef15 in term 0.
I20260929 16:42:20.274138 8761 leader_election.cc:302] T 3b8e4b10e4794f71a7cf01bdb13eefef P ef398814e1ff47409b81c7fd8253ef15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e017281552241b99a107a7afd58b230, ef398814e1ff47409b81c7fd8253ef15; no voters:
I20260929 16:42:20.274612 8910 raft_consensus.cc:2781] T 3b8e4b10e4794f71a7cf01bdb13eefef P ef398814e1ff47409b81c7fd8253ef15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:42:20.274725 8910 raft_consensus.cc:482] T 3b8e4b10e4794f71a7cf01bdb13eefef P ef398814e1ff47409b81c7fd8253ef15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:42:20.274843 8910 raft_consensus.cc:3037] T 3b8e4b10e4794f71a7cf01bdb13eefef P ef398814e1ff47409b81c7fd8253ef15 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:42:20.278467 8910 raft_consensus.cc:504] T 3b8e4b10e4794f71a7cf01bdb13eefef P ef398814e1ff47409b81c7fd8253ef15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } } peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } }
I20260929 16:42:20.280072 8689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b8e4b10e4794f71a7cf01bdb13eefef" candidate_uuid: "ef398814e1ff47409b81c7fd8253ef15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e017281552241b99a107a7afd58b230"
I20260929 16:42:20.280347 8689 raft_consensus.cc:3037] T 3b8e4b10e4794f71a7cf01bdb13eefef P 2e017281552241b99a107a7afd58b230 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:42:20.283787 8689 raft_consensus.cc:2454] T 3b8e4b10e4794f71a7cf01bdb13eefef P 2e017281552241b99a107a7afd58b230 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef398814e1ff47409b81c7fd8253ef15 in term 1.
I20260929 16:42:20.284459 8761 leader_election.cc:302] T 3b8e4b10e4794f71a7cf01bdb13eefef P ef398814e1ff47409b81c7fd8253ef15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e017281552241b99a107a7afd58b230, ef398814e1ff47409b81c7fd8253ef15; no voters:
I20260929 16:42:20.284957 8910 leader_election.cc:288] T 3b8e4b10e4794f71a7cf01bdb13eefef P ef398814e1ff47409b81c7fd8253ef15 [CANDIDATE]: Term 1 election: Requested vote from peers a6d0ac696d0e42ad8808c989facfaee0 (127.0.197.65:44769), 2e017281552241b99a107a7afd58b230 (127.0.197.66:42265)
I20260929 16:42:20.285208 8910 raft_consensus.cc:2781] T 3b8e4b10e4794f71a7cf01bdb13eefef P ef398814e1ff47409b81c7fd8253ef15 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:42:20.285422 8910 raft_consensus.cc:686] T 3b8e4b10e4794f71a7cf01bdb13eefef P ef398814e1ff47409b81c7fd8253ef15 [term 1 LEADER]: Becoming Leader. State: Replica: ef398814e1ff47409b81c7fd8253ef15, State: Running, Role: LEADER
I20260929 16:42:20.286022 8910 consensus_queue.cc:237] T 3b8e4b10e4794f71a7cf01bdb13eefef P ef398814e1ff47409b81c7fd8253ef15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } } peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } }
I20260929 16:42:20.286865 8899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:20.249957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:46129 (local address 127.0.197.66:42265)
0929 16:42:20.270601 (+ 20644us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:20.270610 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:20.270636 (+ 26us) server_negotiation.cc:407] Connection header received
0929 16:42:20.270717 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:20.270725 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:20.270832 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:20.270969 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:42:20.273215 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:20.274767 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:20.284102 (+ 9335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:20.285504 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:20.285674 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:20.285752 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:20.286352 (+ 600us) server_negotiation.cc:299] Negotiation successful
0929 16:42:20.286522 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20478,"thread_start_us":22126,"threads_started":1}
I20260929 16:42:20.287673 8948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:20.249010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.66:42265 (local address 127.0.197.65:46129)
0929 16:42:20.268599 (+ 19589us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:20.268630 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:20.268803 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:20.272226 (+ 3423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:20.272231 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:20.272272 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:42:20.272894 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:20.272914 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:20.275143 (+ 2229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:20.275152 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:20.277830 (+ 2678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:20.277851 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:20.285943 (+ 8092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:20.285949 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:20.286132 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:20.286166 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 16:42:20.287158 (+ 992us) client_negotiation.cc:240] Negotiation successful
0929 16:42:20.287331 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19418,"thread_start_us":27183,"threads_started":1}
I20260929 16:42:20.288380 8689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b8e4b10e4794f71a7cf01bdb13eefef" candidate_uuid: "a6d0ac696d0e42ad8808c989facfaee0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e017281552241b99a107a7afd58b230" is_pre_election: true
I20260929 16:42:20.288820 8689 raft_consensus.cc:2379] T 3b8e4b10e4794f71a7cf01bdb13eefef P 2e017281552241b99a107a7afd58b230 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a6d0ac696d0e42ad8808c989facfaee0 in current term 1: Already voted for candidate ef398814e1ff47409b81c7fd8253ef15 in this term.
I20260929 16:42:20.290691 8947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:20.249559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.67:40929 (local address 127.0.197.65:52977)
0929 16:42:20.275931 (+ 26372us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:20.275952 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:20.276113 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:20.278615 (+ 2502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:20.278620 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:20.278653 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:42:20.279147 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:20.279160 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:20.280842 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:20.280848 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:20.283810 (+ 2962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:20.283831 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:20.289817 (+ 5986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:20.289824 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:20.290010 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:20.290049 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 16:42:20.290133 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 16:42:20.290300 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26213,"thread_start_us":18701,"threads_started":1}
I20260929 16:42:20.291594 8950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:20.250004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:52977 (local address 127.0.197.67:40929)
0929 16:42:20.272016 (+ 22012us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:20.272026 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:20.278022 (+ 5996us) server_negotiation.cc:407] Connection header received
0929 16:42:20.278099 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:20.278105 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:20.278190 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:20.278308 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:42:20.279402 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:20.280660 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:20.287997 (+ 7337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:20.289373 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:20.289567 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:20.289646 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:20.291034 (+ 1388us) server_negotiation.cc:299] Negotiation successful
0929 16:42:20.291204 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21862,"thread_start_us":14747,"threads_started":1}
I20260929 16:42:20.292423 8834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b8e4b10e4794f71a7cf01bdb13eefef" candidate_uuid: "a6d0ac696d0e42ad8808c989facfaee0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef398814e1ff47409b81c7fd8253ef15" is_pre_election: true
I20260929 16:42:20.293974 8944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:20.244514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:44769 (local address 127.0.197.67:56519)
0929 16:42:20.245042 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:20.245065 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:20.245263 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:20.246719 (+ 1456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:20.246725 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:20.246755 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:42:20.247097 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:20.247113 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:20.249175 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:20.249182 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:20.259740 (+ 10558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:20.259778 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:20.293042 (+ 33264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:20.293049 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:20.293289 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:20.293326 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 16:42:20.293437 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 16:42:20.293579 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":231,"threads_started":1}
I20260929 16:42:20.294240 8479 leader_election.cc:302] T 3b8e4b10e4794f71a7cf01bdb13eefef P a6d0ac696d0e42ad8808c989facfaee0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a6d0ac696d0e42ad8808c989facfaee0; no voters: 2e017281552241b99a107a7afd58b230, ef398814e1ff47409b81c7fd8253ef15
I20260929 16:42:20.295086 8912 raft_consensus.cc:3037] T 3b8e4b10e4794f71a7cf01bdb13eefef P a6d0ac696d0e42ad8808c989facfaee0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:42:20.295164 8945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:20.244682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:56519 (local address 127.0.197.65:44769)
0929 16:42:20.246322 (+ 1640us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:20.246331 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:20.246355 (+ 24us) server_negotiation.cc:407] Connection header received
0929 16:42:20.246435 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:20.246440 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:20.246521 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:20.247327 (+ 806us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:42:20.247576 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:20.248993 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:20.265380 (+ 16387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:20.292737 (+ 27357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:20.292948 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:20.293044 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:20.294582 (+ 1538us) server_negotiation.cc:299] Negotiation successful
0929 16:42:20.294760 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":960,"thread_start_us":834,"threads_started":1}
I20260929 16:42:20.295917 8553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b8e4b10e4794f71a7cf01bdb13eefef" candidate_uuid: "ef398814e1ff47409b81c7fd8253ef15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6d0ac696d0e42ad8808c989facfaee0" is_pre_election: true
I20260929 16:42:20.296324 8554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b8e4b10e4794f71a7cf01bdb13eefef" candidate_uuid: "ef398814e1ff47409b81c7fd8253ef15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6d0ac696d0e42ad8808c989facfaee0"
I20260929 16:42:20.298297 8912 raft_consensus.cc:2726] T 3b8e4b10e4794f71a7cf01bdb13eefef P a6d0ac696d0e42ad8808c989facfaee0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 16:42:20.298905 8553 raft_consensus.cc:2454] T 3b8e4b10e4794f71a7cf01bdb13eefef P a6d0ac696d0e42ad8808c989facfaee0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef398814e1ff47409b81c7fd8253ef15 in term 1.
I20260929 16:42:20.300125 8391 catalog_manager.cc:5495] T 3b8e4b10e4794f71a7cf01bdb13eefef P ef398814e1ff47409b81c7fd8253ef15 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef398814e1ff47409b81c7fd8253ef15 (127.0.197.67). New cstate: current_term: 1 leader_uuid: "ef398814e1ff47409b81c7fd8253ef15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } health_report { overall_health: HEALTHY } } }
I20260929 16:42:20.300304 8951 tablet_replica.cc:439] T 3b8e4b10e4794f71a7cf01bdb13eefef P ef398814e1ff47409b81c7fd8253ef15: TxnStatusTablet state changed. Reason: New leader ef398814e1ff47409b81c7fd8253ef15. Latest consensus state: current_term: 1 leader_uuid: "ef398814e1ff47409b81c7fd8253ef15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } } peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } } }
I20260929 16:42:20.300702 8951 tablet_replica.cc:442] T 3b8e4b10e4794f71a7cf01bdb13eefef P ef398814e1ff47409b81c7fd8253ef15: This TxnStatusTablet replica's current role is: LEADER
I20260929 16:42:20.305020 8953 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 16:42:20.305305 8953 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 16:42:20.379137 8950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:20.372788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53394 (local address 127.0.197.67:40929)
0929 16:42:20.373412 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:20.373417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:20.373432 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:42:20.373509 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:20.373512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:20.373589 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:20.373760 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:42:20.374148 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:20.374930 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:20.376536 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:20.376968 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:20.377115 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:20.377841 (+ 726us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:42:20.378428 (+ 587us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:42:20.378603 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:20.378687 (+ 84us) server_negotiation.cc:299] Negotiation successful
0929 16:42:20.378782 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522}
I20260929 16:42:20.379194 8939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:20.372710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.67:40929 (local address 127.0.0.1:53394)
0929 16:42:20.373035 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:20.373053 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:20.373184 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:20.373777 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:20.373782 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:20.373794 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:42:20.374005 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:20.374014 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:20.375115 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:20.375121 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:20.376375 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:20.376390 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:20.377385 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:20.377390 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:20.377555 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:20.377617 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:42:20.378553 (+ 936us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:42:20.378575 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 16:42:20.378666 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 16:42:20.378838 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145}
I20260929 16:42:20.385036 8553 raft_consensus.cc:1261] T 3b8e4b10e4794f71a7cf01bdb13eefef P a6d0ac696d0e42ad8808c989facfaee0 [term 1 FOLLOWER]: Refusing update from remote peer ef398814e1ff47409b81c7fd8253ef15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:42:20.385959 8689 raft_consensus.cc:1261] T 3b8e4b10e4794f71a7cf01bdb13eefef P 2e017281552241b99a107a7afd58b230 [term 1 FOLLOWER]: Refusing update from remote peer ef398814e1ff47409b81c7fd8253ef15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:42:20.385918 8951 consensus_queue.cc:1035] T 3b8e4b10e4794f71a7cf01bdb13eefef P ef398814e1ff47409b81c7fd8253ef15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:42:20.386678 8951 consensus_queue.cc:1035] T 3b8e4b10e4794f71a7cf01bdb13eefef P ef398814e1ff47409b81c7fd8253ef15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:42:20.389676 8912 tablet_replica.cc:439] T 3b8e4b10e4794f71a7cf01bdb13eefef P a6d0ac696d0e42ad8808c989facfaee0: TxnStatusTablet state changed. Reason: New leader ef398814e1ff47409b81c7fd8253ef15. Latest consensus state: current_term: 1 leader_uuid: "ef398814e1ff47409b81c7fd8253ef15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } } peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } } }
I20260929 16:42:20.390000 8912 tablet_replica.cc:442] T 3b8e4b10e4794f71a7cf01bdb13eefef P a6d0ac696d0e42ad8808c989facfaee0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:42:20.394245 8957 mvcc.cc:205] Tried to move back new op lower bound from 7334707775005483008 to 7334707774635012096. Current Snapshot: MvccSnapshot[applied={T|T < 7334707775005483008}]
I20260929 16:42:20.395176 8956 mvcc.cc:205] Tried to move back new op lower bound from 7334707775005483008 to 7334707774635012096. Current Snapshot: MvccSnapshot[applied={T|T < 7334707775005483008 or (T in {7334707775005483008})}]
I20260929 16:42:20.396077 8912 tablet_replica.cc:439] T 3b8e4b10e4794f71a7cf01bdb13eefef P a6d0ac696d0e42ad8808c989facfaee0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ef398814e1ff47409b81c7fd8253ef15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } } peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } } }
I20260929 16:42:20.396375 8912 tablet_replica.cc:442] T 3b8e4b10e4794f71a7cf01bdb13eefef P a6d0ac696d0e42ad8808c989facfaee0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:42:20.396818 8951 tablet_replica.cc:439] T 3b8e4b10e4794f71a7cf01bdb13eefef P ef398814e1ff47409b81c7fd8253ef15: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ef398814e1ff47409b81c7fd8253ef15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } } peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } } }
I20260929 16:42:20.398600 8951 tablet_replica.cc:442] T 3b8e4b10e4794f71a7cf01bdb13eefef P ef398814e1ff47409b81c7fd8253ef15: This TxnStatusTablet replica's current role is: LEADER
I20260929 16:42:20.400892 8911 tablet_replica.cc:439] T 3b8e4b10e4794f71a7cf01bdb13eefef P 2e017281552241b99a107a7afd58b230: TxnStatusTablet state changed. Reason: New leader ef398814e1ff47409b81c7fd8253ef15. Latest consensus state: current_term: 1 leader_uuid: "ef398814e1ff47409b81c7fd8253ef15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } } peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } } }
I20260929 16:42:20.401171 8911 tablet_replica.cc:442] T 3b8e4b10e4794f71a7cf01bdb13eefef P 2e017281552241b99a107a7afd58b230: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:42:20.402840 8951 tablet_replica.cc:439] T 3b8e4b10e4794f71a7cf01bdb13eefef P ef398814e1ff47409b81c7fd8253ef15: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ef398814e1ff47409b81c7fd8253ef15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } } peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } } }
I20260929 16:42:20.403126 8951 tablet_replica.cc:442] T 3b8e4b10e4794f71a7cf01bdb13eefef P ef398814e1ff47409b81c7fd8253ef15: This TxnStatusTablet replica's current role is: LEADER
I20260929 16:42:20.407842 8915 tablet_replica.cc:439] T 3b8e4b10e4794f71a7cf01bdb13eefef P 2e017281552241b99a107a7afd58b230: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ef398814e1ff47409b81c7fd8253ef15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } } peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } } }
I20260929 16:42:20.408202 8915 tablet_replica.cc:442] T 3b8e4b10e4794f71a7cf01bdb13eefef P 2e017281552241b99a107a7afd58b230: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 16:42:20.408381 8915 consensus_queue.cc:1035] T 25a691dde75740d0aa9167e6ff021195 P 2e017281552241b99a107a7afd58b230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:42:20.418249 8915 consensus_queue.cc:1035] T 25a691dde75740d0aa9167e6ff021195 P 2e017281552241b99a107a7afd58b230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:42:20.434602 8950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:20.426841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:37275 (local address 127.0.197.67:40929)
0929 16:42:20.427035 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:20.427043 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:20.427150 (+ 107us) server_negotiation.cc:407] Connection header received
0929 16:42:20.427501 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:20.427508 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:20.427606 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:20.427689 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:42:20.428514 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:20.430045 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:20.432274 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:20.432957 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:20.433062 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:20.433767 (+ 705us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:42:20.433915 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:42:20.433976 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:20.434191 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 16:42:20.434281 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 16:42:20.435406 8968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:20.426333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.67:40929 (local address 127.0.197.66:37275)
0929 16:42:20.426997 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:20.427055 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:20.427317 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:20.427939 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:20.427946 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:20.427964 (+ 18us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:42:20.428304 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:20.428315 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:20.430231 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:20.430236 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:20.432055 (+ 1819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:20.432074 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:20.433285 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:20.433291 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:20.433475 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:20.433557 (+ 82us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:42:20.434051 (+ 494us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:42:20.434061 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 16:42:20.434808 (+ 747us) client_negotiation.cc:240] Negotiation successful
0929 16:42:20.434983 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":718,"threads_started":1}
I20260929 16:42:20.481503 8899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:20.473465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:51905 (local address 127.0.197.66:42265)
0929 16:42:20.473691 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:20.473701 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:20.474341 (+ 640us) server_negotiation.cc:407] Connection header received
0929 16:42:20.474812 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:20.474818 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:20.474904 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:20.475032 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:42:20.475715 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:20.476551 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:20.478983 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:20.479927 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:20.480086 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:20.480662 (+ 576us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:42:20.480817 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:42:20.480909 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:20.481067 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 16:42:20.481163 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 16:42:20.481644 8972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:20.473443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.66:42265 (local address 127.0.197.67:51905)
0929 16:42:20.474219 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:20.474238 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:20.474411 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:20.475089 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:20.475098 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:20.475112 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:42:20.475528 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:20.475550 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:20.476690 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:20.476697 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:20.478632 (+ 1935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:20.478653 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:20.480169 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:20.480176 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:20.480330 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:20.480431 (+ 101us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:42:20.480925 (+ 494us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:42:20.480943 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 16:42:20.481022 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 16:42:20.481173 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":403,"threads_started":1}
I20260929 16:42:20.526079 8392 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:42:20.526357 8392 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:42:20.529819 8392 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a62329d5a3f14fc2846e3e1a948f8aa9: Sending DeleteTablet for 3 replicas of tablet 25a691dde75740d0aa9167e6ff021195
I20260929 16:42:20.530807 8529 tablet_service.cc:1514] Processing DeleteTablet for tablet 25a691dde75740d0aa9167e6ff021195 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:42:20 UTC) from {username='oryx', principal='oryx/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:51312
I20260929 16:42:20.530925 8814 tablet_service.cc:1514] Processing DeleteTablet for tablet 25a691dde75740d0aa9167e6ff021195 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:42:20 UTC) from {username='oryx', principal='oryx/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:53382
I20260929 16:42:20.531733 8668 tablet_service.cc:1514] Processing DeleteTablet for tablet 25a691dde75740d0aa9167e6ff021195 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:42:20 UTC) from {username='oryx', principal='oryx/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:58764
I20260929 16:42:20.540069 8974 tablet_replica.cc:330] T 25a691dde75740d0aa9167e6ff021195 P a6d0ac696d0e42ad8808c989facfaee0: stopping tablet replica
I20260929 16:42:20.540369 8974 raft_consensus.cc:2229] T 25a691dde75740d0aa9167e6ff021195 P a6d0ac696d0e42ad8808c989facfaee0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:42:20.540658 8974 raft_consensus.cc:2258] T 25a691dde75740d0aa9167e6ff021195 P a6d0ac696d0e42ad8808c989facfaee0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:42:20.545135 8975 tablet_replica.cc:330] T 25a691dde75740d0aa9167e6ff021195 P ef398814e1ff47409b81c7fd8253ef15: stopping tablet replica
I20260929 16:42:20.545403 8975 raft_consensus.cc:2229] T 25a691dde75740d0aa9167e6ff021195 P ef398814e1ff47409b81c7fd8253ef15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:42:20.545720 8975 raft_consensus.cc:2258] T 25a691dde75740d0aa9167e6ff021195 P ef398814e1ff47409b81c7fd8253ef15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:42:20.546413 8976 tablet_replica.cc:330] T 25a691dde75740d0aa9167e6ff021195 P 2e017281552241b99a107a7afd58b230: stopping tablet replica
I20260929 16:42:20.546635 8976 raft_consensus.cc:2229] T 25a691dde75740d0aa9167e6ff021195 P 2e017281552241b99a107a7afd58b230 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:42:20.547160 8976 raft_consensus.cc:2258] T 25a691dde75740d0aa9167e6ff021195 P 2e017281552241b99a107a7afd58b230 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:42:20.548301 8975 ts_tablet_manager.cc:1905] T 25a691dde75740d0aa9167e6ff021195 P ef398814e1ff47409b81c7fd8253ef15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:42:20.548329 8974 ts_tablet_manager.cc:1905] T 25a691dde75740d0aa9167e6ff021195 P a6d0ac696d0e42ad8808c989facfaee0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:42:20.548691 8976 ts_tablet_manager.cc:1905] T 25a691dde75740d0aa9167e6ff021195 P 2e017281552241b99a107a7afd58b230: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:42:20.556310 8974 ts_tablet_manager.cc:1918] T 25a691dde75740d0aa9167e6ff021195 P a6d0ac696d0e42ad8808c989facfaee0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:42:20.556579 8974 log.cc:1196] T 25a691dde75740d0aa9167e6ff021195 P a6d0ac696d0e42ad8808c989facfaee0: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-0/wal/wals/25a691dde75740d0aa9167e6ff021195
I20260929 16:42:20.557137 8974 ts_tablet_manager.cc:1939] T 25a691dde75740d0aa9167e6ff021195 P a6d0ac696d0e42ad8808c989facfaee0: Deleting consensus metadata
I20260929 16:42:20.557413 8975 ts_tablet_manager.cc:1918] T 25a691dde75740d0aa9167e6ff021195 P ef398814e1ff47409b81c7fd8253ef15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:42:20.557675 8975 log.cc:1196] T 25a691dde75740d0aa9167e6ff021195 P ef398814e1ff47409b81c7fd8253ef15: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-2/wal/wals/25a691dde75740d0aa9167e6ff021195
I20260929 16:42:20.558514 8975 ts_tablet_manager.cc:1939] T 25a691dde75740d0aa9167e6ff021195 P ef398814e1ff47409b81c7fd8253ef15: Deleting consensus metadata
I20260929 16:42:20.558617 8378 catalog_manager.cc:4841] TS a6d0ac696d0e42ad8808c989facfaee0 (127.0.197.65:44769): tablet 25a691dde75740d0aa9167e6ff021195 (table test-table [id=c6d9f02fc7224144bf53b1ae17b26cd0]) successfully deleted
I20260929 16:42:20.560437 8376 catalog_manager.cc:4841] TS ef398814e1ff47409b81c7fd8253ef15 (127.0.197.67:40929): tablet 25a691dde75740d0aa9167e6ff021195 (table test-table [id=c6d9f02fc7224144bf53b1ae17b26cd0]) successfully deleted
I20260929 16:42:20.566562 8976 ts_tablet_manager.cc:1918] T 25a691dde75740d0aa9167e6ff021195 P 2e017281552241b99a107a7afd58b230: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:42:20.566723 8976 log.cc:1196] T 25a691dde75740d0aa9167e6ff021195 P 2e017281552241b99a107a7afd58b230: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-1/wal/wals/25a691dde75740d0aa9167e6ff021195
I20260929 16:42:20.567265 8976 ts_tablet_manager.cc:1939] T 25a691dde75740d0aa9167e6ff021195 P 2e017281552241b99a107a7afd58b230: Deleting consensus metadata
I20260929 16:42:20.568557 8378 catalog_manager.cc:4841] TS 2e017281552241b99a107a7afd58b230 (127.0.197.66:42265): tablet 25a691dde75740d0aa9167e6ff021195 (table test-table [id=c6d9f02fc7224144bf53b1ae17b26cd0]) successfully deleted
Sep 29 16:42:20 dist-test-slave-l31f krb5kdc[8347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 16:42:20 dist-test-slave-l31f krb5kdc[8347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:42:20 dist-test-slave-l31f krb5kdc[8347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
W20260929 16:42:20.604180 8983 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:42:20.590390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54194 (local address 127.0.197.126:38787)
0929 16:42:20.591471 (+ 1081us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:20.591478 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:20.591821 (+ 343us) server_negotiation.cc:407] Connection header received
0929 16:42:20.592099 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:20.592105 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:20.592193 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:20.592329 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:20.593760 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:20.594922 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:20.596340 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:20.597033 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:20.597176 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:20.603572 (+ 6396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54194: BlockingRecv error: recv got EOF from 127.0.0.1:54194 (error 108)
Metrics: {"server-negotiator.queue_time_us":885,"thread_start_us":782,"threads_started":1}
Sep 29 16:42:20 dist-test-slave-l31f krb5kdc[8347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM
I20260929 16:42:20.625679 8983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:20.606867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54202 (local address 127.0.197.126:38787)
0929 16:42:20.607072 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:20.607079 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:20.607786 (+ 707us) server_negotiation.cc:407] Connection header received
0929 16:42:20.608023 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:20.608028 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:20.608116 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:20.608215 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:20.609724 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:20.610788 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:20.613079 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:20.613577 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:20.613749 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:20.618750 (+ 5001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:20.618758 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:20.618763 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:20.618818 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:20.622497 (+ 3679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:20.623457 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:20.623464 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:20.623468 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:20.623664 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:20.624050 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:20.624055 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:20.624060 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:20.624436 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:20.624719 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:20.625051 (+ 332us) server_negotiation.cc:299] Negotiation successful
0929 16:42:20.625321 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 16:42:20.630354 8392 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54202:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:42:20.631145 8392 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:42:20.638500 8529 tablet_service.cc:1467] Processing CreateTablet for tablet aecb6c630645480cb1a8766c48a47ff8 (DEFAULT_TABLE table=test-table [id=493d869720c445adbff9b2f066cf2826]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:42:20.638728 8668 tablet_service.cc:1467] Processing CreateTablet for tablet aecb6c630645480cb1a8766c48a47ff8 (DEFAULT_TABLE table=test-table [id=493d869720c445adbff9b2f066cf2826]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:42:20.638787 8814 tablet_service.cc:1467] Processing CreateTablet for tablet aecb6c630645480cb1a8766c48a47ff8 (DEFAULT_TABLE table=test-table [id=493d869720c445adbff9b2f066cf2826]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:42:20.638979 8529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aecb6c630645480cb1a8766c48a47ff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:42:20.639187 8814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aecb6c630645480cb1a8766c48a47ff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:42:20.639209 8668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aecb6c630645480cb1a8766c48a47ff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:42:20.651134 8990 tablet_bootstrap.cc:492] T aecb6c630645480cb1a8766c48a47ff8 P 2e017281552241b99a107a7afd58b230: Bootstrap starting.
I20260929 16:42:20.654318 8992 tablet_bootstrap.cc:492] T aecb6c630645480cb1a8766c48a47ff8 P ef398814e1ff47409b81c7fd8253ef15: Bootstrap starting.
I20260929 16:42:20.654536 8990 tablet_bootstrap.cc:654] T aecb6c630645480cb1a8766c48a47ff8 P 2e017281552241b99a107a7afd58b230: Neither blocks nor log segments found. Creating new log.
I20260929 16:42:20.655155 8991 tablet_bootstrap.cc:492] T aecb6c630645480cb1a8766c48a47ff8 P a6d0ac696d0e42ad8808c989facfaee0: Bootstrap starting.
I20260929 16:42:20.659348 8991 tablet_bootstrap.cc:654] T aecb6c630645480cb1a8766c48a47ff8 P a6d0ac696d0e42ad8808c989facfaee0: Neither blocks nor log segments found. Creating new log.
I20260929 16:42:20.668226 8992 tablet_bootstrap.cc:654] T aecb6c630645480cb1a8766c48a47ff8 P ef398814e1ff47409b81c7fd8253ef15: Neither blocks nor log segments found. Creating new log.
I20260929 16:42:20.677192 8991 tablet_bootstrap.cc:492] T aecb6c630645480cb1a8766c48a47ff8 P a6d0ac696d0e42ad8808c989facfaee0: No bootstrap required, opened a new log
I20260929 16:42:20.677474 8991 ts_tablet_manager.cc:1397] T aecb6c630645480cb1a8766c48a47ff8 P a6d0ac696d0e42ad8808c989facfaee0: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.007s
I20260929 16:42:20.678399 8991 raft_consensus.cc:348] T aecb6c630645480cb1a8766c48a47ff8 P a6d0ac696d0e42ad8808c989facfaee0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } } peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } }
I20260929 16:42:20.678824 8991 raft_consensus.cc:374] T aecb6c630645480cb1a8766c48a47ff8 P a6d0ac696d0e42ad8808c989facfaee0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:42:20.678970 8991 raft_consensus.cc:729] T aecb6c630645480cb1a8766c48a47ff8 P a6d0ac696d0e42ad8808c989facfaee0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6d0ac696d0e42ad8808c989facfaee0, State: Initialized, Role: FOLLOWER
I20260929 16:42:20.679390 8991 consensus_queue.cc:260] T aecb6c630645480cb1a8766c48a47ff8 P a6d0ac696d0e42ad8808c989facfaee0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } } peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } }
I20260929 16:42:20.680389 8991 ts_tablet_manager.cc:1428] T aecb6c630645480cb1a8766c48a47ff8 P a6d0ac696d0e42ad8808c989facfaee0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 16:42:20.682432 8992 tablet_bootstrap.cc:492] T aecb6c630645480cb1a8766c48a47ff8 P ef398814e1ff47409b81c7fd8253ef15: No bootstrap required, opened a new log
I20260929 16:42:20.682670 8992 ts_tablet_manager.cc:1397] T aecb6c630645480cb1a8766c48a47ff8 P ef398814e1ff47409b81c7fd8253ef15: Time spent bootstrapping tablet: real 0.028s user 0.007s sys 0.000s
I20260929 16:42:20.683040 8990 tablet_bootstrap.cc:492] T aecb6c630645480cb1a8766c48a47ff8 P 2e017281552241b99a107a7afd58b230: No bootstrap required, opened a new log
I20260929 16:42:20.683248 8990 ts_tablet_manager.cc:1397] T aecb6c630645480cb1a8766c48a47ff8 P 2e017281552241b99a107a7afd58b230: Time spent bootstrapping tablet: real 0.032s user 0.007s sys 0.023s
I20260929 16:42:20.683799 8992 raft_consensus.cc:348] T aecb6c630645480cb1a8766c48a47ff8 P ef398814e1ff47409b81c7fd8253ef15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } } peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } }
I20260929 16:42:20.684170 8992 raft_consensus.cc:374] T aecb6c630645480cb1a8766c48a47ff8 P ef398814e1ff47409b81c7fd8253ef15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:42:20.684090 8990 raft_consensus.cc:348] T aecb6c630645480cb1a8766c48a47ff8 P 2e017281552241b99a107a7afd58b230 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } } peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } }
I20260929 16:42:20.684396 8990 raft_consensus.cc:374] T aecb6c630645480cb1a8766c48a47ff8 P 2e017281552241b99a107a7afd58b230 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:42:20.684336 8992 raft_consensus.cc:729] T aecb6c630645480cb1a8766c48a47ff8 P ef398814e1ff47409b81c7fd8253ef15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef398814e1ff47409b81c7fd8253ef15, State: Initialized, Role: FOLLOWER
I20260929 16:42:20.684540 8990 raft_consensus.cc:729] T aecb6c630645480cb1a8766c48a47ff8 P 2e017281552241b99a107a7afd58b230 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e017281552241b99a107a7afd58b230, State: Initialized, Role: FOLLOWER
I20260929 16:42:20.684728 8992 consensus_queue.cc:260] T aecb6c630645480cb1a8766c48a47ff8 P ef398814e1ff47409b81c7fd8253ef15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } } peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } }
I20260929 16:42:20.684818 8990 consensus_queue.cc:260] T aecb6c630645480cb1a8766c48a47ff8 P 2e017281552241b99a107a7afd58b230 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } } peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } }
I20260929 16:42:20.685541 8990 ts_tablet_manager.cc:1428] T aecb6c630645480cb1a8766c48a47ff8 P 2e017281552241b99a107a7afd58b230: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 16:42:20.686618 8992 ts_tablet_manager.cc:1428] T aecb6c630645480cb1a8766c48a47ff8 P ef398814e1ff47409b81c7fd8253ef15: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260929 16:42:20.783679 8911 raft_consensus.cc:482] T aecb6c630645480cb1a8766c48a47ff8 P 2e017281552241b99a107a7afd58b230 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:42:20.783936 8911 raft_consensus.cc:504] T aecb6c630645480cb1a8766c48a47ff8 P 2e017281552241b99a107a7afd58b230 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } } peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } }
I20260929 16:42:20.784693 8911 leader_election.cc:288] T aecb6c630645480cb1a8766c48a47ff8 P 2e017281552241b99a107a7afd58b230 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6d0ac696d0e42ad8808c989facfaee0 (127.0.197.65:44769), ef398814e1ff47409b81c7fd8253ef15 (127.0.197.67:40929)
I20260929 16:42:20.785424 8554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aecb6c630645480cb1a8766c48a47ff8" candidate_uuid: "2e017281552241b99a107a7afd58b230" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6d0ac696d0e42ad8808c989facfaee0" is_pre_election: true
I20260929 16:42:20.785444 8834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aecb6c630645480cb1a8766c48a47ff8" candidate_uuid: "2e017281552241b99a107a7afd58b230" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef398814e1ff47409b81c7fd8253ef15" is_pre_election: true
I20260929 16:42:20.785734 8554 raft_consensus.cc:2454] T aecb6c630645480cb1a8766c48a47ff8 P a6d0ac696d0e42ad8808c989facfaee0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e017281552241b99a107a7afd58b230 in term 0.
I20260929 16:42:20.785753 8834 raft_consensus.cc:2454] T aecb6c630645480cb1a8766c48a47ff8 P ef398814e1ff47409b81c7fd8253ef15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e017281552241b99a107a7afd58b230 in term 0.
I20260929 16:42:20.786378 8621 leader_election.cc:302] T aecb6c630645480cb1a8766c48a47ff8 P 2e017281552241b99a107a7afd58b230 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2e017281552241b99a107a7afd58b230, a6d0ac696d0e42ad8808c989facfaee0, ef398814e1ff47409b81c7fd8253ef15; no voters:
I20260929 16:42:20.786744 8911 raft_consensus.cc:2781] T aecb6c630645480cb1a8766c48a47ff8 P 2e017281552241b99a107a7afd58b230 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:42:20.786851 8911 raft_consensus.cc:482] T aecb6c630645480cb1a8766c48a47ff8 P 2e017281552241b99a107a7afd58b230 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:42:20.786943 8911 raft_consensus.cc:3037] T aecb6c630645480cb1a8766c48a47ff8 P 2e017281552241b99a107a7afd58b230 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:42:20.790072 8911 raft_consensus.cc:504] T aecb6c630645480cb1a8766c48a47ff8 P 2e017281552241b99a107a7afd58b230 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } } peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } }
I20260929 16:42:20.790822 8911 leader_election.cc:288] T aecb6c630645480cb1a8766c48a47ff8 P 2e017281552241b99a107a7afd58b230 [CANDIDATE]: Term 1 election: Requested vote from peers a6d0ac696d0e42ad8808c989facfaee0 (127.0.197.65:44769), ef398814e1ff47409b81c7fd8253ef15 (127.0.197.67:40929)
I20260929 16:42:20.791441 8554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aecb6c630645480cb1a8766c48a47ff8" candidate_uuid: "2e017281552241b99a107a7afd58b230" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6d0ac696d0e42ad8808c989facfaee0"
I20260929 16:42:20.791469 8834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aecb6c630645480cb1a8766c48a47ff8" candidate_uuid: "2e017281552241b99a107a7afd58b230" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef398814e1ff47409b81c7fd8253ef15"
I20260929 16:42:20.791700 8554 raft_consensus.cc:3037] T aecb6c630645480cb1a8766c48a47ff8 P a6d0ac696d0e42ad8808c989facfaee0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:42:20.791709 8834 raft_consensus.cc:3037] T aecb6c630645480cb1a8766c48a47ff8 P ef398814e1ff47409b81c7fd8253ef15 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:42:20.794283 8834 raft_consensus.cc:2454] T aecb6c630645480cb1a8766c48a47ff8 P ef398814e1ff47409b81c7fd8253ef15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e017281552241b99a107a7afd58b230 in term 1.
I20260929 16:42:20.794814 8554 raft_consensus.cc:2454] T aecb6c630645480cb1a8766c48a47ff8 P a6d0ac696d0e42ad8808c989facfaee0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e017281552241b99a107a7afd58b230 in term 1.
I20260929 16:42:20.795003 8619 leader_election.cc:302] T aecb6c630645480cb1a8766c48a47ff8 P 2e017281552241b99a107a7afd58b230 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e017281552241b99a107a7afd58b230, ef398814e1ff47409b81c7fd8253ef15; no voters:
I20260929 16:42:20.795431 8911 raft_consensus.cc:2781] T aecb6c630645480cb1a8766c48a47ff8 P 2e017281552241b99a107a7afd58b230 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:42:20.795665 8911 raft_consensus.cc:686] T aecb6c630645480cb1a8766c48a47ff8 P 2e017281552241b99a107a7afd58b230 [term 1 LEADER]: Becoming Leader. State: Replica: 2e017281552241b99a107a7afd58b230, State: Running, Role: LEADER
I20260929 16:42:20.795934 8911 consensus_queue.cc:237] T aecb6c630645480cb1a8766c48a47ff8 P 2e017281552241b99a107a7afd58b230 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } } peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } }
I20260929 16:42:20.799052 8391 catalog_manager.cc:5495] T aecb6c630645480cb1a8766c48a47ff8 P 2e017281552241b99a107a7afd58b230 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e017281552241b99a107a7afd58b230 (127.0.197.66). New cstate: current_term: 1 leader_uuid: "2e017281552241b99a107a7afd58b230" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } health_report { overall_health: UNKNOWN } } }
I20260929 16:42:20.819965 8899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:20.814920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58790 (local address 127.0.197.66:42265)
0929 16:42:20.815190 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:20.815198 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:20.815225 (+ 27us) server_negotiation.cc:407] Connection header received
0929 16:42:20.815477 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:20.815483 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:20.815563 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:20.815670 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:42:20.816026 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:20.816768 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:20.818139 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:20.818587 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:20.818735 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:20.819232 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:42:20.819346 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:42:20.819452 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:20.819598 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 16:42:20.819685 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260929 16:42:20.823223 8554 raft_consensus.cc:1261] T aecb6c630645480cb1a8766c48a47ff8 P a6d0ac696d0e42ad8808c989facfaee0 [term 1 FOLLOWER]: Refusing update from remote peer 2e017281552241b99a107a7afd58b230: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:42:20.823222 8834 raft_consensus.cc:1261] T aecb6c630645480cb1a8766c48a47ff8 P ef398814e1ff47409b81c7fd8253ef15 [term 1 FOLLOWER]: Refusing update from remote peer 2e017281552241b99a107a7afd58b230: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:42:20.823966 8911 consensus_queue.cc:1035] T aecb6c630645480cb1a8766c48a47ff8 P 2e017281552241b99a107a7afd58b230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:42:20.824297 8970 consensus_queue.cc:1035] T aecb6c630645480cb1a8766c48a47ff8 P 2e017281552241b99a107a7afd58b230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:42:20.838132 8392 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:42:20.838414 8392 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:42:20.841528 8392 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a62329d5a3f14fc2846e3e1a948f8aa9: Sending DeleteTablet for 3 replicas of tablet aecb6c630645480cb1a8766c48a47ff8
I20260929 16:42:20.842630 8529 tablet_service.cc:1514] Processing DeleteTablet for tablet aecb6c630645480cb1a8766c48a47ff8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:42:20 UTC) from {username='oryx', principal='oryx/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:51312
I20260929 16:42:20.842598 8668 tablet_service.cc:1514] Processing DeleteTablet for tablet aecb6c630645480cb1a8766c48a47ff8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:42:20 UTC) from {username='oryx', principal='oryx/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:58764
I20260929 16:42:20.842789 8814 tablet_service.cc:1514] Processing DeleteTablet for tablet aecb6c630645480cb1a8766c48a47ff8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:42:20 UTC) from {username='oryx', principal='oryx/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:53382
I20260929 16:42:20.842932 8974 tablet_replica.cc:330] T aecb6c630645480cb1a8766c48a47ff8 P a6d0ac696d0e42ad8808c989facfaee0: stopping tablet replica
I20260929 16:42:20.843125 8975 tablet_replica.cc:330] T aecb6c630645480cb1a8766c48a47ff8 P ef398814e1ff47409b81c7fd8253ef15: stopping tablet replica
I20260929 16:42:20.843156 8974 raft_consensus.cc:2229] T aecb6c630645480cb1a8766c48a47ff8 P a6d0ac696d0e42ad8808c989facfaee0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:42:20.843329 8975 raft_consensus.cc:2229] T aecb6c630645480cb1a8766c48a47ff8 P ef398814e1ff47409b81c7fd8253ef15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:42:20.843364 8976 tablet_replica.cc:330] T aecb6c630645480cb1a8766c48a47ff8 P 2e017281552241b99a107a7afd58b230: stopping tablet replica
I20260929 16:42:20.843417 8974 raft_consensus.cc:2258] T aecb6c630645480cb1a8766c48a47ff8 P a6d0ac696d0e42ad8808c989facfaee0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:42:20.843578 8975 raft_consensus.cc:2258] T aecb6c630645480cb1a8766c48a47ff8 P ef398814e1ff47409b81c7fd8253ef15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:42:20.843588 8976 raft_consensus.cc:2229] T aecb6c630645480cb1a8766c48a47ff8 P 2e017281552241b99a107a7afd58b230 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:42:20.843947 8976 raft_consensus.cc:2258] T aecb6c630645480cb1a8766c48a47ff8 P 2e017281552241b99a107a7afd58b230 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:42:20.844620 8974 ts_tablet_manager.cc:1905] T aecb6c630645480cb1a8766c48a47ff8 P a6d0ac696d0e42ad8808c989facfaee0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:42:20.844712 8975 ts_tablet_manager.cc:1905] T aecb6c630645480cb1a8766c48a47ff8 P ef398814e1ff47409b81c7fd8253ef15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:42:20.845175 8976 ts_tablet_manager.cc:1905] T aecb6c630645480cb1a8766c48a47ff8 P 2e017281552241b99a107a7afd58b230: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:42:20.845130 8392 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54202:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:42:20.845882 8392 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:42:20.851995 8976 ts_tablet_manager.cc:1918] T aecb6c630645480cb1a8766c48a47ff8 P 2e017281552241b99a107a7afd58b230: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:42:20.852154 8974 ts_tablet_manager.cc:1918] T aecb6c630645480cb1a8766c48a47ff8 P a6d0ac696d0e42ad8808c989facfaee0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:42:20.852255 8976 log.cc:1196] T aecb6c630645480cb1a8766c48a47ff8 P 2e017281552241b99a107a7afd58b230: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-1/wal/wals/aecb6c630645480cb1a8766c48a47ff8
I20260929 16:42:20.852385 8974 log.cc:1196] T aecb6c630645480cb1a8766c48a47ff8 P a6d0ac696d0e42ad8808c989facfaee0: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-0/wal/wals/aecb6c630645480cb1a8766c48a47ff8
I20260929 16:42:20.852916 8974 ts_tablet_manager.cc:1939] T aecb6c630645480cb1a8766c48a47ff8 P a6d0ac696d0e42ad8808c989facfaee0: Deleting consensus metadata
I20260929 16:42:20.852932 8976 ts_tablet_manager.cc:1939] T aecb6c630645480cb1a8766c48a47ff8 P 2e017281552241b99a107a7afd58b230: Deleting consensus metadata
I20260929 16:42:20.854313 8378 catalog_manager.cc:4841] TS 2e017281552241b99a107a7afd58b230 (127.0.197.66:42265): tablet aecb6c630645480cb1a8766c48a47ff8 (table test-table [id=493d869720c445adbff9b2f066cf2826]) successfully deleted
I20260929 16:42:20.854836 8378 catalog_manager.cc:4841] TS a6d0ac696d0e42ad8808c989facfaee0 (127.0.197.65:44769): tablet aecb6c630645480cb1a8766c48a47ff8 (table test-table [id=493d869720c445adbff9b2f066cf2826]) successfully deleted
I20260929 16:42:20.854851 8814 tablet_service.cc:1467] Processing CreateTablet for tablet e58ca26c6e3c437b8b7dccc7f5e24d0f (DEFAULT_TABLE table=test-table [id=5319a2937c8c4932bd65de611c623340]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:42:20.855353 8814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e58ca26c6e3c437b8b7dccc7f5e24d0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:42:20.855357 8529 tablet_service.cc:1467] Processing CreateTablet for tablet e58ca26c6e3c437b8b7dccc7f5e24d0f (DEFAULT_TABLE table=test-table [id=5319a2937c8c4932bd65de611c623340]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:42:20.855786 8529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e58ca26c6e3c437b8b7dccc7f5e24d0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:42:20.859548 8668 tablet_service.cc:1467] Processing CreateTablet for tablet e58ca26c6e3c437b8b7dccc7f5e24d0f (DEFAULT_TABLE table=test-table [id=5319a2937c8c4932bd65de611c623340]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:42:20.860064 8668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e58ca26c6e3c437b8b7dccc7f5e24d0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:42:20.861147 8975 ts_tablet_manager.cc:1918] T aecb6c630645480cb1a8766c48a47ff8 P ef398814e1ff47409b81c7fd8253ef15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:42:20.861781 8991 tablet_bootstrap.cc:492] T e58ca26c6e3c437b8b7dccc7f5e24d0f P a6d0ac696d0e42ad8808c989facfaee0: Bootstrap starting.
I20260929 16:42:20.861862 8992 tablet_bootstrap.cc:492] T e58ca26c6e3c437b8b7dccc7f5e24d0f P ef398814e1ff47409b81c7fd8253ef15: Bootstrap starting.
I20260929 16:42:20.863852 8991 tablet_bootstrap.cc:654] T e58ca26c6e3c437b8b7dccc7f5e24d0f P a6d0ac696d0e42ad8808c989facfaee0: Neither blocks nor log segments found. Creating new log.
I20260929 16:42:20.863899 8975 log.cc:1196] T aecb6c630645480cb1a8766c48a47ff8 P ef398814e1ff47409b81c7fd8253ef15: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-2/wal/wals/aecb6c630645480cb1a8766c48a47ff8
I20260929 16:42:20.863969 8992 tablet_bootstrap.cc:654] T e58ca26c6e3c437b8b7dccc7f5e24d0f P ef398814e1ff47409b81c7fd8253ef15: Neither blocks nor log segments found. Creating new log.
I20260929 16:42:20.864504 8975 ts_tablet_manager.cc:1939] T aecb6c630645480cb1a8766c48a47ff8 P ef398814e1ff47409b81c7fd8253ef15: Deleting consensus metadata
I20260929 16:42:20.865957 8376 catalog_manager.cc:4841] TS ef398814e1ff47409b81c7fd8253ef15 (127.0.197.67:40929): tablet aecb6c630645480cb1a8766c48a47ff8 (table test-table [id=493d869720c445adbff9b2f066cf2826]) successfully deleted
I20260929 16:42:20.867944 8992 tablet_bootstrap.cc:492] T e58ca26c6e3c437b8b7dccc7f5e24d0f P ef398814e1ff47409b81c7fd8253ef15: No bootstrap required, opened a new log
I20260929 16:42:20.867949 8991 tablet_bootstrap.cc:492] T e58ca26c6e3c437b8b7dccc7f5e24d0f P a6d0ac696d0e42ad8808c989facfaee0: No bootstrap required, opened a new log
I20260929 16:42:20.868141 8991 ts_tablet_manager.cc:1397] T e58ca26c6e3c437b8b7dccc7f5e24d0f P a6d0ac696d0e42ad8808c989facfaee0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 16:42:20.868137 8992 ts_tablet_manager.cc:1397] T e58ca26c6e3c437b8b7dccc7f5e24d0f P ef398814e1ff47409b81c7fd8253ef15: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260929 16:42:20.868954 8992 raft_consensus.cc:348] T e58ca26c6e3c437b8b7dccc7f5e24d0f P ef398814e1ff47409b81c7fd8253ef15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } } peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } }
I20260929 16:42:20.869010 8991 raft_consensus.cc:348] T e58ca26c6e3c437b8b7dccc7f5e24d0f P a6d0ac696d0e42ad8808c989facfaee0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } } peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } }
I20260929 16:42:20.869248 8992 raft_consensus.cc:374] T e58ca26c6e3c437b8b7dccc7f5e24d0f P ef398814e1ff47409b81c7fd8253ef15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:42:20.869272 8991 raft_consensus.cc:374] T e58ca26c6e3c437b8b7dccc7f5e24d0f P a6d0ac696d0e42ad8808c989facfaee0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:42:20.869351 8992 raft_consensus.cc:729] T e58ca26c6e3c437b8b7dccc7f5e24d0f P ef398814e1ff47409b81c7fd8253ef15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef398814e1ff47409b81c7fd8253ef15, State: Initialized, Role: FOLLOWER
I20260929 16:42:20.869351 8991 raft_consensus.cc:729] T e58ca26c6e3c437b8b7dccc7f5e24d0f P a6d0ac696d0e42ad8808c989facfaee0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6d0ac696d0e42ad8808c989facfaee0, State: Initialized, Role: FOLLOWER
I20260929 16:42:20.869611 8991 consensus_queue.cc:260] T e58ca26c6e3c437b8b7dccc7f5e24d0f P a6d0ac696d0e42ad8808c989facfaee0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } } peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } }
I20260929 16:42:20.869611 8992 consensus_queue.cc:260] T e58ca26c6e3c437b8b7dccc7f5e24d0f P ef398814e1ff47409b81c7fd8253ef15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } } peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } }
I20260929 16:42:20.870180 8991 ts_tablet_manager.cc:1428] T e58ca26c6e3c437b8b7dccc7f5e24d0f P a6d0ac696d0e42ad8808c989facfaee0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 16:42:20.870195 8992 ts_tablet_manager.cc:1428] T e58ca26c6e3c437b8b7dccc7f5e24d0f P ef398814e1ff47409b81c7fd8253ef15: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 16:42:20.870384 8990 tablet_bootstrap.cc:492] T e58ca26c6e3c437b8b7dccc7f5e24d0f P 2e017281552241b99a107a7afd58b230: Bootstrap starting.
I20260929 16:42:20.872622 8990 tablet_bootstrap.cc:654] T e58ca26c6e3c437b8b7dccc7f5e24d0f P 2e017281552241b99a107a7afd58b230: Neither blocks nor log segments found. Creating new log.
I20260929 16:42:20.875159 8990 tablet_bootstrap.cc:492] T e58ca26c6e3c437b8b7dccc7f5e24d0f P 2e017281552241b99a107a7afd58b230: No bootstrap required, opened a new log
I20260929 16:42:20.875376 8990 ts_tablet_manager.cc:1397] T e58ca26c6e3c437b8b7dccc7f5e24d0f P 2e017281552241b99a107a7afd58b230: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260929 16:42:20.876235 8990 raft_consensus.cc:348] T e58ca26c6e3c437b8b7dccc7f5e24d0f P 2e017281552241b99a107a7afd58b230 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } } peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } }
I20260929 16:42:20.876502 8990 raft_consensus.cc:374] T e58ca26c6e3c437b8b7dccc7f5e24d0f P 2e017281552241b99a107a7afd58b230 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:42:20.876616 8990 raft_consensus.cc:729] T e58ca26c6e3c437b8b7dccc7f5e24d0f P 2e017281552241b99a107a7afd58b230 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e017281552241b99a107a7afd58b230, State: Initialized, Role: FOLLOWER
I20260929 16:42:20.876916 8990 consensus_queue.cc:260] T e58ca26c6e3c437b8b7dccc7f5e24d0f P 2e017281552241b99a107a7afd58b230 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } } peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } }
I20260929 16:42:20.877530 8990 ts_tablet_manager.cc:1428] T e58ca26c6e3c437b8b7dccc7f5e24d0f P 2e017281552241b99a107a7afd58b230: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 16:42:21.058651 8969 raft_consensus.cc:482] T e58ca26c6e3c437b8b7dccc7f5e24d0f P ef398814e1ff47409b81c7fd8253ef15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:42:21.058935 8969 raft_consensus.cc:504] T e58ca26c6e3c437b8b7dccc7f5e24d0f P ef398814e1ff47409b81c7fd8253ef15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } } peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } }
I20260929 16:42:21.059675 8969 leader_election.cc:288] T e58ca26c6e3c437b8b7dccc7f5e24d0f P ef398814e1ff47409b81c7fd8253ef15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6d0ac696d0e42ad8808c989facfaee0 (127.0.197.65:44769), 2e017281552241b99a107a7afd58b230 (127.0.197.66:42265)
I20260929 16:42:21.060256 8554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e58ca26c6e3c437b8b7dccc7f5e24d0f" candidate_uuid: "ef398814e1ff47409b81c7fd8253ef15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6d0ac696d0e42ad8808c989facfaee0" is_pre_election: true
I20260929 16:42:21.060387 8688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e58ca26c6e3c437b8b7dccc7f5e24d0f" candidate_uuid: "ef398814e1ff47409b81c7fd8253ef15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e017281552241b99a107a7afd58b230" is_pre_election: true
I20260929 16:42:21.060564 8554 raft_consensus.cc:2454] T e58ca26c6e3c437b8b7dccc7f5e24d0f P a6d0ac696d0e42ad8808c989facfaee0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef398814e1ff47409b81c7fd8253ef15 in term 0.
I20260929 16:42:21.060619 8688 raft_consensus.cc:2454] T e58ca26c6e3c437b8b7dccc7f5e24d0f P 2e017281552241b99a107a7afd58b230 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef398814e1ff47409b81c7fd8253ef15 in term 0.
I20260929 16:42:21.061015 8761 leader_election.cc:302] T e58ca26c6e3c437b8b7dccc7f5e24d0f P ef398814e1ff47409b81c7fd8253ef15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6d0ac696d0e42ad8808c989facfaee0, ef398814e1ff47409b81c7fd8253ef15; no voters:
I20260929 16:42:21.061370 8969 raft_consensus.cc:2781] T e58ca26c6e3c437b8b7dccc7f5e24d0f P ef398814e1ff47409b81c7fd8253ef15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:42:21.061509 8969 raft_consensus.cc:482] T e58ca26c6e3c437b8b7dccc7f5e24d0f P ef398814e1ff47409b81c7fd8253ef15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:42:21.061630 8969 raft_consensus.cc:3037] T e58ca26c6e3c437b8b7dccc7f5e24d0f P ef398814e1ff47409b81c7fd8253ef15 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:42:21.063905 8969 raft_consensus.cc:504] T e58ca26c6e3c437b8b7dccc7f5e24d0f P ef398814e1ff47409b81c7fd8253ef15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } } peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } }
I20260929 16:42:21.064595 8969 leader_election.cc:288] T e58ca26c6e3c437b8b7dccc7f5e24d0f P ef398814e1ff47409b81c7fd8253ef15 [CANDIDATE]: Term 1 election: Requested vote from peers a6d0ac696d0e42ad8808c989facfaee0 (127.0.197.65:44769), 2e017281552241b99a107a7afd58b230 (127.0.197.66:42265)
I20260929 16:42:21.065223 8554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e58ca26c6e3c437b8b7dccc7f5e24d0f" candidate_uuid: "ef398814e1ff47409b81c7fd8253ef15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6d0ac696d0e42ad8808c989facfaee0"
I20260929 16:42:21.065268 8688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e58ca26c6e3c437b8b7dccc7f5e24d0f" candidate_uuid: "ef398814e1ff47409b81c7fd8253ef15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e017281552241b99a107a7afd58b230"
I20260929 16:42:21.065502 8688 raft_consensus.cc:3037] T e58ca26c6e3c437b8b7dccc7f5e24d0f P 2e017281552241b99a107a7afd58b230 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:42:21.065516 8554 raft_consensus.cc:3037] T e58ca26c6e3c437b8b7dccc7f5e24d0f P a6d0ac696d0e42ad8808c989facfaee0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:42:21.067842 8688 raft_consensus.cc:2454] T e58ca26c6e3c437b8b7dccc7f5e24d0f P 2e017281552241b99a107a7afd58b230 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef398814e1ff47409b81c7fd8253ef15 in term 1.
I20260929 16:42:21.068068 8554 raft_consensus.cc:2454] T e58ca26c6e3c437b8b7dccc7f5e24d0f P a6d0ac696d0e42ad8808c989facfaee0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef398814e1ff47409b81c7fd8253ef15 in term 1.
I20260929 16:42:21.068423 8761 leader_election.cc:302] T e58ca26c6e3c437b8b7dccc7f5e24d0f P ef398814e1ff47409b81c7fd8253ef15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e017281552241b99a107a7afd58b230, ef398814e1ff47409b81c7fd8253ef15; no voters:
I20260929 16:42:21.068840 8969 raft_consensus.cc:2781] T e58ca26c6e3c437b8b7dccc7f5e24d0f P ef398814e1ff47409b81c7fd8253ef15 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:42:21.069279 8969 raft_consensus.cc:686] T e58ca26c6e3c437b8b7dccc7f5e24d0f P ef398814e1ff47409b81c7fd8253ef15 [term 1 LEADER]: Becoming Leader. State: Replica: ef398814e1ff47409b81c7fd8253ef15, State: Running, Role: LEADER
I20260929 16:42:21.069586 8969 consensus_queue.cc:237] T e58ca26c6e3c437b8b7dccc7f5e24d0f P ef398814e1ff47409b81c7fd8253ef15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } } peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } }
I20260929 16:42:21.072499 8391 catalog_manager.cc:5495] T e58ca26c6e3c437b8b7dccc7f5e24d0f P ef398814e1ff47409b81c7fd8253ef15 reported cstate change: term changed from 0 to 1, leader changed from <none> to ef398814e1ff47409b81c7fd8253ef15 (127.0.197.67). New cstate: current_term: 1 leader_uuid: "ef398814e1ff47409b81c7fd8253ef15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef398814e1ff47409b81c7fd8253ef15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 40929 } health_report { overall_health: HEALTHY } } }
I20260929 16:42:21.126799 9006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:21.120767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53410 (local address 127.0.197.67:40929)
0929 16:42:21.121437 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:21.121442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:21.121459 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:42:21.121517 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:21.121520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:21.121594 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:21.121725 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:42:21.122193 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:21.122935 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:21.124530 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:21.125007 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:21.125156 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:21.125983 (+ 827us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:42:21.126120 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:42:21.126202 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:21.126448 (+ 246us) server_negotiation.cc:299] Negotiation successful
0929 16:42:21.126518 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":243,"threads_started":1}
I20260929 16:42:21.141727 9006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:21.136047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:33047 (local address 127.0.197.67:40929)
0929 16:42:21.136256 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:21.136266 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:21.136668 (+ 402us) server_negotiation.cc:407] Connection header received
0929 16:42:21.136812 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:21.136815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:21.136877 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:21.136955 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:42:21.137403 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:21.138274 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:21.139869 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:21.140518 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:21.140635 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:21.141066 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:42:21.141173 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:42:21.141246 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:21.141398 (+ 152us) server_negotiation.cc:299] Negotiation successful
0929 16:42:21.141462 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":56}
I20260929 16:42:21.141773 9009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:21.135995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.67:40929 (local address 127.0.197.67:33047)
0929 16:42:21.136566 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:21.136582 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:21.136702 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:21.136983 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:21.136986 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:21.137011 (+ 25us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 16:42:21.137253 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:21.137261 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:21.138433 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:21.138436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:21.139605 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:21.139621 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:21.140716 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:21.140721 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:21.140875 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:21.140915 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 16:42:21.141261 (+ 346us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 16:42:21.141269 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 16:42:21.141334 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 16:42:21.141449 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":17,"thread_start_us":340,"threads_started":1}
I20260929 16:42:21.151710 8688 raft_consensus.cc:1261] T e58ca26c6e3c437b8b7dccc7f5e24d0f P 2e017281552241b99a107a7afd58b230 [term 1 FOLLOWER]: Refusing update from remote peer ef398814e1ff47409b81c7fd8253ef15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:42:21.151832 8554 raft_consensus.cc:1261] T e58ca26c6e3c437b8b7dccc7f5e24d0f P a6d0ac696d0e42ad8808c989facfaee0 [term 1 FOLLOWER]: Refusing update from remote peer ef398814e1ff47409b81c7fd8253ef15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:42:21.152364 9005 consensus_queue.cc:1035] T e58ca26c6e3c437b8b7dccc7f5e24d0f P ef398814e1ff47409b81c7fd8253ef15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e017281552241b99a107a7afd58b230" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:42:21.152712 9003 consensus_queue.cc:1035] T e58ca26c6e3c437b8b7dccc7f5e24d0f P ef398814e1ff47409b81c7fd8253ef15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6d0ac696d0e42ad8808c989facfaee0" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 44769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:42:21.211727 8391 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:42:21.212040 8391 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:42:21.215536 8391 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a62329d5a3f14fc2846e3e1a948f8aa9: Sending DeleteTablet for 3 replicas of tablet e58ca26c6e3c437b8b7dccc7f5e24d0f
I20260929 16:42:21.216476 8814 tablet_service.cc:1514] Processing DeleteTablet for tablet e58ca26c6e3c437b8b7dccc7f5e24d0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:42:21 UTC) from {username='oryx', principal='oryx/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:53382
I20260929 16:42:21.216801 8975 tablet_replica.cc:330] T e58ca26c6e3c437b8b7dccc7f5e24d0f P ef398814e1ff47409b81c7fd8253ef15: stopping tablet replica
I20260929 16:42:21.216787 8529 tablet_service.cc:1514] Processing DeleteTablet for tablet e58ca26c6e3c437b8b7dccc7f5e24d0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:42:21 UTC) from {username='oryx', principal='oryx/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:51312
I20260929 16:42:21.217021 8975 raft_consensus.cc:2229] T e58ca26c6e3c437b8b7dccc7f5e24d0f P ef398814e1ff47409b81c7fd8253ef15 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:42:21.217118 8668 tablet_service.cc:1514] Processing DeleteTablet for tablet e58ca26c6e3c437b8b7dccc7f5e24d0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:42:21 UTC) from {username='oryx', principal='oryx/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:58764
I20260929 16:42:21.217440 8974 tablet_replica.cc:330] T e58ca26c6e3c437b8b7dccc7f5e24d0f P a6d0ac696d0e42ad8808c989facfaee0: stopping tablet replica
I20260929 16:42:21.217587 8975 raft_consensus.cc:2258] T e58ca26c6e3c437b8b7dccc7f5e24d0f P ef398814e1ff47409b81c7fd8253ef15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:42:21.217638 8974 raft_consensus.cc:2229] T e58ca26c6e3c437b8b7dccc7f5e24d0f P a6d0ac696d0e42ad8808c989facfaee0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:42:21.217842 8974 raft_consensus.cc:2258] T e58ca26c6e3c437b8b7dccc7f5e24d0f P a6d0ac696d0e42ad8808c989facfaee0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:42:21.218129 8976 tablet_replica.cc:330] T e58ca26c6e3c437b8b7dccc7f5e24d0f P 2e017281552241b99a107a7afd58b230: stopping tablet replica
I20260929 16:42:21.218320 8976 raft_consensus.cc:2229] T e58ca26c6e3c437b8b7dccc7f5e24d0f P 2e017281552241b99a107a7afd58b230 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:42:21.218530 8976 raft_consensus.cc:2258] T e58ca26c6e3c437b8b7dccc7f5e24d0f P 2e017281552241b99a107a7afd58b230 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:42:21.218979 8975 ts_tablet_manager.cc:1905] T e58ca26c6e3c437b8b7dccc7f5e24d0f P ef398814e1ff47409b81c7fd8253ef15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:42:21.219480 8974 ts_tablet_manager.cc:1905] T e58ca26c6e3c437b8b7dccc7f5e24d0f P a6d0ac696d0e42ad8808c989facfaee0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:42:21.224390 8975 ts_tablet_manager.cc:1918] T e58ca26c6e3c437b8b7dccc7f5e24d0f P ef398814e1ff47409b81c7fd8253ef15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:42:21.224570 8975 log.cc:1196] T e58ca26c6e3c437b8b7dccc7f5e24d0f P ef398814e1ff47409b81c7fd8253ef15: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-2/wal/wals/e58ca26c6e3c437b8b7dccc7f5e24d0f
I20260929 16:42:21.225021 8975 ts_tablet_manager.cc:1939] T e58ca26c6e3c437b8b7dccc7f5e24d0f P ef398814e1ff47409b81c7fd8253ef15: Deleting consensus metadata
I20260929 16:42:21.225544 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 8463
I20260929 16:42:21.227524 8376 catalog_manager.cc:4841] TS ef398814e1ff47409b81c7fd8253ef15 (127.0.197.67:40929): tablet e58ca26c6e3c437b8b7dccc7f5e24d0f (table test-table [id=5319a2937c8c4932bd65de611c623340]) successfully deleted
I20260929 16:42:21.228788 8974 ts_tablet_manager.cc:1918] T e58ca26c6e3c437b8b7dccc7f5e24d0f P a6d0ac696d0e42ad8808c989facfaee0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:42:21.228943 8974 log.cc:1196] T e58ca26c6e3c437b8b7dccc7f5e24d0f P a6d0ac696d0e42ad8808c989facfaee0: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-0/wal/wals/e58ca26c6e3c437b8b7dccc7f5e24d0f
I20260929 16:42:21.229393 8974 ts_tablet_manager.cc:1939] T e58ca26c6e3c437b8b7dccc7f5e24d0f P a6d0ac696d0e42ad8808c989facfaee0: Deleting consensus metadata
I20260929 16:42:21.230754 8378 catalog_manager.cc:4841] TS a6d0ac696d0e42ad8808c989facfaee0 (127.0.197.65:44769): tablet e58ca26c6e3c437b8b7dccc7f5e24d0f (table test-table [id=5319a2937c8c4932bd65de611c623340]) successfully deleted
I20260929 16:42:21.231299 8976 ts_tablet_manager.cc:1905] T e58ca26c6e3c437b8b7dccc7f5e24d0f P 2e017281552241b99a107a7afd58b230: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:42:21.238771 8976 ts_tablet_manager.cc:1918] T e58ca26c6e3c437b8b7dccc7f5e24d0f P 2e017281552241b99a107a7afd58b230: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 16:42:21.239025 8976 log.cc:1196] T e58ca26c6e3c437b8b7dccc7f5e24d0f P 2e017281552241b99a107a7afd58b230: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790700068255611-789-0/minicluster-data/ts-1/wal/wals/e58ca26c6e3c437b8b7dccc7f5e24d0f
I20260929 16:42:21.239494 8976 ts_tablet_manager.cc:1939] T e58ca26c6e3c437b8b7dccc7f5e24d0f P 2e017281552241b99a107a7afd58b230: Deleting consensus metadata
I20260929 16:42:21.240818 8378 catalog_manager.cc:4841] TS 2e017281552241b99a107a7afd58b230 (127.0.197.66:42265): tablet e58ca26c6e3c437b8b7dccc7f5e24d0f (table test-table [id=5319a2937c8c4932bd65de611c623340]) successfully deleted
Sep 29 16:42:21 dist-test-slave-l31f krb5kdc[8347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.197.65@KRBTEST.COM
I20260929 16:42:21.251483 9015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:21.226505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51322 (local address 127.0.197.65:44769)
0929 16:42:21.227353 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:21.227362 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:21.236132 (+ 8770us) server_negotiation.cc:407] Connection header received
0929 16:42:21.236344 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:21.236348 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:21.236427 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:21.236549 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:21.237751 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:21.238455 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:21.239951 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:21.240380 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:21.240547 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:21.244677 (+ 4130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:21.244696 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:21.244714 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:21.244754 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:21.248232 (+ 3478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:21.249059 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:21.249067 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:21.249069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:21.249217 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:21.249617 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:21.249626 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:21.249628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:21.250004 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:21.250307 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:21.250730 (+ 423us) server_negotiation.cc:299] Negotiation successful
0929 16:42:21.251038 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":556,"threads_started":1}
I20260929 16:42:21.256217 8594 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:42:21.441648 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 8463
I20260929 16:42:21.469048 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 8603
Sep 29 16:42:21 dist-test-slave-l31f krb5kdc[8347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.197.66@KRBTEST.COM
I20260929 16:42:21.486932 9018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:21.469905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58794 (local address 127.0.197.66:42265)
0929 16:42:21.470414 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:21.470418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:21.470433 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:42:21.470486 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:21.470490 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:21.470545 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:21.470673 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:21.472073 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:21.472831 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:21.474248 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:21.474710 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:21.474859 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:21.480573 (+ 5714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:21.480590 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:21.480608 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:21.480657 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:21.483941 (+ 3284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:21.484749 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:21.484763 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:21.484765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:21.484900 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:21.485291 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:21.485298 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:21.485301 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:21.485667 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:21.485991 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:21.486329 (+ 338us) server_negotiation.cc:299] Negotiation successful
0929 16:42:21.486591 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":252,"threads_started":1}
I20260929 16:42:21.487704 8734 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:42:21.654084 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 8603
I20260929 16:42:21.679679 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 8743
Sep 29 16:42:21 dist-test-slave-l31f krb5kdc[8347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.197.67@KRBTEST.COM
I20260929 16:42:21.701749 9020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:21.680752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53412 (local address 127.0.197.67:40929)
0929 16:42:21.681309 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:21.681313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:21.681330 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:42:21.681387 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:21.681390 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:21.681447 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:21.681547 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:21.683591 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:21.684433 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:21.686490 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:21.686970 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:21.687182 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:21.693041 (+ 5859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:21.693058 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:21.693075 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:21.693120 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:21.696951 (+ 3831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:21.698090 (+ 1139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:21.698104 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:21.698110 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:21.698382 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:21.699128 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:21.699136 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:21.699139 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:21.700111 (+ 972us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:21.700475 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:21.701016 (+ 541us) server_negotiation.cc:299] Negotiation successful
0929 16:42:21.701316 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":307,"threads_started":1}
I20260929 16:42:21.702806 8874 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:42:21.859956 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 8743
I20260929 16:42:21.885824 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 8360
I20260929 16:42:21.887157 8432 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:42:22.003973 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 8360
2026-09-29T16:42:22Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (7529 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790700068255611-789-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:42:22 dist-test-slave-l31f krb5kdc[9026](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:42:22 dist-test-slave-l31f krb5kdc[9026](info): set up 2 sockets
Sep 29 16:42:22 dist-test-slave-l31f krb5kdc[9026](info): commencing operation
krb5kdc: starting...
W20260929 16:42:23.128155 789 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790700068255611-789-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790700068255611-789-0/krb5kdc/test-user.keytab.
Sep 29 16:42:23 dist-test-slave-l31f krb5kdc[9026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:42:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:42:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.197.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
I20260929 16:42:23.376720 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:40279
--webserver_interface=127.0.197.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:45599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.126:40279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:42:23.725087 9039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:42:23.725502 9039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:42:23.725672 9039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:42:23.733307 9039 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:42:23.733505 9039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:42:23.733625 9039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:42:23.733734 9039 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:42:23.733826 9039 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:42:23.744262 9039 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:45599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790700068255611-789-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.197.126:40279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:40279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.197.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:42:23.745690 9039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:42:23.747109 9039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:42:23.759822 9047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:23.760030 9045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:23.760087 9044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:42:23.760888 9039 server_base.cc:1034] running on GCE node
I20260929 16:42:23.761724 9039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:42:23.763386 9039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:42:23.764663 9039 hybrid_clock.cc:648] HybridClock initialized: now 1790700143764596 us; error 56 us; skew 500 ppm
I20260929 16:42:23.765105 9039 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-29T16:42:23Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1831 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:42:23 dist-test-slave-l31f krb5kdc[9053](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:42:23 dist-test-slave-l31f krb5kdc[9053](info): set up 2 sockets
Sep 29 16:42:23 dist-test-slave-l31f krb5kdc[9053](info): commencing operation
krb5kdc: starting...
W20260929 16:42:24.956322 789 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.006s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/test-user.keytab.
Sep 29 16:42:25 dist-test-slave-l31f krb5kdc[9053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:42:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:42:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.197.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
I20260929 16:42:25.215723 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:41429
--webserver_interface=127.0.197.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:45967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.126:41429
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:42:25.569334 9066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:42:25.569885 9066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:42:25.570065 9066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:42:25.577844 9066 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:42:25.577992 9066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:42:25.578040 9066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:42:25.578091 9066 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:42:25.578130 9066 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:42:25.588632 9066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:45967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.197.126:41429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:41429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.197.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:42:25.590085 9066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:42:25.591398 9066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:42:25.604751 9072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:25.605285 9074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:25.605566 9071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:42:25.607547 9066 server_base.cc:1034] running on GCE node
I20260929 16:42:25.608434 9066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:42:25.610116 9066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:42:25.611330 9066 hybrid_clock.cc:648] HybridClock initialized: now 1790700145611284 us; error 41 us; skew 500 ppm
I20260929 16:42:25.611729 9066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:42:25 dist-test-slave-l31f krb5kdc[9053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:42:25.619925 9066 init.cc:377] Logged in from keytab as kudu/127.0.197.126@KRBTEST.COM (short username kudu)
I20260929 16:42:25.621755 9066 webserver.cc:460] Webserver started at http://127.0.197.126:37763/ using document root <none> and password file <none>
I20260929 16:42:25.622546 9066 fs_manager.cc:359] Metadata directory not provided
I20260929 16:42:25.622664 9066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:42:25.623131 9066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:42:25.626322 9066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/master-0/data/instance:
uuid: "4feb71ed104840ebbeb9474b650b5e78"
format_stamp: "Formatted at 2026-09-29 16:42:25 on dist-test-slave-l31f"
server_key: "de3bb815d20f5a5d02cf04eaf3ae1d97"
server_key_iv: "f86ac5e0b15175fd69583bd6839a5861"
server_key_version: "encryptionkey@0"
I20260929 16:42:25.627439 9066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/master-0/wal/instance:
uuid: "4feb71ed104840ebbeb9474b650b5e78"
format_stamp: "Formatted at 2026-09-29 16:42:25 on dist-test-slave-l31f"
server_key: "de3bb815d20f5a5d02cf04eaf3ae1d97"
server_key_iv: "f86ac5e0b15175fd69583bd6839a5861"
server_key_version: "encryptionkey@0"
I20260929 16:42:25.637460 9066 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.000s sys 0.008s
I20260929 16:42:25.643918 9081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:25.646483 9066 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260929 16:42:25.646728 9066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "4feb71ed104840ebbeb9474b650b5e78"
format_stamp: "Formatted at 2026-09-29 16:42:25 on dist-test-slave-l31f"
server_key: "de3bb815d20f5a5d02cf04eaf3ae1d97"
server_key_iv: "f86ac5e0b15175fd69583bd6839a5861"
server_key_version: "encryptionkey@0"
I20260929 16:42:25.647080 9066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:42:25.668344 9066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:42:25.675076 9066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:42:25.675477 9066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:42:25.696264 9066 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.126:41429
I20260929 16:42:25.696287 9132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.126:41429 every 8 connection(s)
I20260929 16:42:25.697669 9066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:42:25.701400 9133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:42:25.702173 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 9066
I20260929 16:42:25.702389 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/master-0/wal/instance
I20260929 16:42:25.703059 789 external_mini_cluster.cc:1428] Setting key f411923ff825707728e52ec0d98437bd
I20260929 16:42:25.716204 9133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4feb71ed104840ebbeb9474b650b5e78: Bootstrap starting.
Sep 29 16:42:25 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:42:25.720553 9133 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4feb71ed104840ebbeb9474b650b5e78: Neither blocks nor log segments found. Creating new log.
I20260929 16:42:25.721726 9133 log.cc:824] T 00000000000000000000000000000000 P 4feb71ed104840ebbeb9474b650b5e78: Log is configured to *not* fsync() on all Append() calls
I20260929 16:42:25.725368 9133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4feb71ed104840ebbeb9474b650b5e78: No bootstrap required, opened a new log
I20260929 16:42:25.729853 9136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:25.705112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53878 (local address 127.0.197.126:41429)
0929 16:42:25.705959 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:25.705973 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:25.706016 (+ 43us) server_negotiation.cc:407] Connection header received
0929 16:42:25.707845 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:25.707876 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:25.708326 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:25.708700 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:25.710615 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:25.712174 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:25.714095 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:25.714877 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:25.715127 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:25.720343 (+ 5216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:25.720353 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:25.720372 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:25.720420 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:25.724409 (+ 3989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:25.725733 (+ 1324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:25.725743 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:25.725752 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:25.725995 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:25.726379 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:25.726384 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:25.726386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:25.727166 (+ 780us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:25.727477 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:25.727895 (+ 418us) server_negotiation.cc:299] Negotiation successful
0929 16:42:25.728383 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":411,"threads_started":1}
I20260929 16:42:25.735145 9133 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4feb71ed104840ebbeb9474b650b5e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4feb71ed104840ebbeb9474b650b5e78" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 41429 } }
I20260929 16:42:25.735555 9133 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4feb71ed104840ebbeb9474b650b5e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:42:25.735658 9133 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4feb71ed104840ebbeb9474b650b5e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4feb71ed104840ebbeb9474b650b5e78, State: Initialized, Role: FOLLOWER
I20260929 16:42:25.736286 9133 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4feb71ed104840ebbeb9474b650b5e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4feb71ed104840ebbeb9474b650b5e78" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 41429 } }
I20260929 16:42:25.736491 9133 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4feb71ed104840ebbeb9474b650b5e78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:42:25.736645 9133 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4feb71ed104840ebbeb9474b650b5e78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:42:25.736828 9133 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4feb71ed104840ebbeb9474b650b5e78 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:42:25.739157 9133 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4feb71ed104840ebbeb9474b650b5e78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4feb71ed104840ebbeb9474b650b5e78" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 41429 } }
I20260929 16:42:25.739734 9133 leader_election.cc:302] T 00000000000000000000000000000000 P 4feb71ed104840ebbeb9474b650b5e78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4feb71ed104840ebbeb9474b650b5e78; no voters:
I20260929 16:42:25.740366 9133 leader_election.cc:288] T 00000000000000000000000000000000 P 4feb71ed104840ebbeb9474b650b5e78 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:42:25.740525 9138 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4feb71ed104840ebbeb9474b650b5e78 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:42:25.741202 9138 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4feb71ed104840ebbeb9474b650b5e78 [term 1 LEADER]: Becoming Leader. State: Replica: 4feb71ed104840ebbeb9474b650b5e78, State: Running, Role: LEADER
I20260929 16:42:25.741853 9138 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4feb71ed104840ebbeb9474b650b5e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4feb71ed104840ebbeb9474b650b5e78" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 41429 } }
I20260929 16:42:25.742628 9133 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4feb71ed104840ebbeb9474b650b5e78 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:42:25.744964 9139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4feb71ed104840ebbeb9474b650b5e78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4feb71ed104840ebbeb9474b650b5e78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4feb71ed104840ebbeb9474b650b5e78" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 41429 } } }
I20260929 16:42:25.745360 9139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4feb71ed104840ebbeb9474b650b5e78 [sys.catalog]: This master's current role is: LEADER
I20260929 16:42:25.745765 9140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4feb71ed104840ebbeb9474b650b5e78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4feb71ed104840ebbeb9474b650b5e78. Latest consensus state: current_term: 1 leader_uuid: "4feb71ed104840ebbeb9474b650b5e78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4feb71ed104840ebbeb9474b650b5e78" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 41429 } } }
I20260929 16:42:25.746049 9140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4feb71ed104840ebbeb9474b650b5e78 [sys.catalog]: This master's current role is: LEADER
I20260929 16:42:25.747920 9145 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:42:25.753194 9145 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:42:25.765894 9145 catalog_manager.cc:1347] Generated new cluster ID: 8b6884fa10a746faae2b91e482af1614
I20260929 16:42:25.766109 9145 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:42:25.775904 9145 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:42:25.777915 9145 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:42:25.789664 9145 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4feb71ed104840ebbeb9474b650b5e78: Generated new TSK 0
I20260929 16:42:25.790874 9145 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.197.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab.
Entry for principal kudu/127.0.197.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab.
I20260929 16:42:25.876811 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.65:0
--local_ip_for_outbound_sockets=127.0.197.65
--webserver_interface=127.0.197.65
--webserver_port=0
--tserver_master_addrs=127.0.197.126:41429
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab
--principal=kudu/127.0.197.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:45967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:42:26.258945 9159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:42:26.259358 9159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:42:26.259459 9159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:42:26.266991 9159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:42:26.267218 9159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:42:26.267359 9159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.65
I20260929 16:42:26.277542 9159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:45967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab
--principal=kudu/127.0.197.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.197.65
--webserver_port=0
--tserver_master_addrs=127.0.197.126:41429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.65
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:42:26.279233 9159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:42:26.280834 9159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:42:26.295993 9164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:26.295993 9165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:26.296047 9167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:42:26.296904 9159 server_base.cc:1034] running on GCE node
I20260929 16:42:26.297650 9159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:42:26.298768 9159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:42:26.300139 9159 hybrid_clock.cc:648] HybridClock initialized: now 1790700146300012 us; error 87 us; skew 500 ppm
I20260929 16:42:26.300751 9159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:42:26 dist-test-slave-l31f krb5kdc[9053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:42:26.304661 9159 init.cc:377] Logged in from keytab as kudu/127.0.197.65@KRBTEST.COM (short username kudu)
I20260929 16:42:26.306640 9159 webserver.cc:460] Webserver started at http://127.0.197.65:42585/ using document root <none> and password file <none>
I20260929 16:42:26.307499 9159 fs_manager.cc:359] Metadata directory not provided
I20260929 16:42:26.307672 9159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:42:26.308017 9159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:42:26.312282 9159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-0/data/instance:
uuid: "ea6bac8f757445e48100b025a25b24e2"
format_stamp: "Formatted at 2026-09-29 16:42:26 on dist-test-slave-l31f"
server_key: "95b7938121eb01788eda5b0a1cbf2808"
server_key_iv: "06a6c2b703564a49f64c6845513eb2cf"
server_key_version: "encryptionkey@0"
I20260929 16:42:26.313660 9159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-0/wal/instance:
uuid: "ea6bac8f757445e48100b025a25b24e2"
format_stamp: "Formatted at 2026-09-29 16:42:26 on dist-test-slave-l31f"
server_key: "95b7938121eb01788eda5b0a1cbf2808"
server_key_iv: "06a6c2b703564a49f64c6845513eb2cf"
server_key_version: "encryptionkey@0"
I20260929 16:42:26.320102 9159 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 16:42:26.324232 9174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:26.325857 9159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:42:26.326082 9159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-0/wal
uuid: "ea6bac8f757445e48100b025a25b24e2"
format_stamp: "Formatted at 2026-09-29 16:42:26 on dist-test-slave-l31f"
server_key: "95b7938121eb01788eda5b0a1cbf2808"
server_key_iv: "06a6c2b703564a49f64c6845513eb2cf"
server_key_version: "encryptionkey@0"
I20260929 16:42:26.326460 9159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:42:26.356662 9159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:42:26.363530 9159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:42:26.363900 9159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:42:26.365057 9159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:42:26.366726 9159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:42:26.366958 9159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:26.367188 9159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:42:26.367270 9159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:26.403599 9159 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.65:34557
I20260929 16:42:26.403631 9286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.65:34557 every 8 connection(s)
I20260929 16:42:26.405035 9159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
I20260929 16:42:26.406952 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 9159
I20260929 16:42:26.407195 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-0/wal/instance
I20260929 16:42:26.407876 789 external_mini_cluster.cc:1428] Setting key bf9db9ab0bc12b52a4f0712036950222
Sep 29 16:42:26 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.65@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:42:26.428661 9289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:26.408044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:60081 (local address 127.0.197.126:41429)
0929 16:42:26.408555 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:26.408561 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:26.410208 (+ 1647us) server_negotiation.cc:407] Connection header received
0929 16:42:26.412131 (+ 1923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:26.412136 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:26.412236 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:26.412362 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:26.413995 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:26.414728 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:26.416931 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:26.417372 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:26.417506 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:26.422328 (+ 4822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:26.422333 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:26.422337 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:26.422376 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:26.425196 (+ 2820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:26.426111 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:26.426116 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:26.426118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:26.426256 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:26.426736 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:26.426752 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:26.426755 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:26.427094 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:26.427252 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:26.428030 (+ 778us) server_negotiation.cc:299] Negotiation successful
0929 16:42:26.428283 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":260,"threads_started":1}
I20260929 16:42:26.429939 9290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:26.408791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:41429 (local address 127.0.197.65:60081)
0929 16:42:26.409776 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:26.409834 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:26.411800 (+ 1966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:26.412632 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:26.412641 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:26.413203 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:26.413799 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:26.413818 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:26.414913 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:26.414918 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:26.416699 (+ 1781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:26.416721 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:26.417647 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:26.417657 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:26.418041 (+ 384us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:26.418930 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:26.418969 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:26.422015 (+ 3046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:42:26.425409 (+ 3394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:26.425420 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:26.425443 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:26.425946 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:26.426383 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:26.426388 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:26.426392 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:26.426610 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:26.427247 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:42:26.427255 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:42:26.427627 (+ 372us) client_negotiation.cc:769] Sending connection context
0929 16:42:26.427916 (+ 289us) client_negotiation.cc:240] Negotiation successful
0929 16:42:26.428489 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":406,"threads_started":1}
WARNING: no policy specified for kudu/127.0.197.66@KRBTEST.COM; defaulting to no policy
I20260929 16:42:26.431960 9287 heartbeater.cc:347] Connected to a master server at 127.0.197.126:41429
I20260929 16:42:26.432301 9287 heartbeater.cc:464] Registering TS with master...
I20260929 16:42:26.433140 9287 heartbeater.cc:511] Master 127.0.197.126:41429 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.66@KRBTEST.COM" created.
I20260929 16:42:26.436234 9098 ts_manager.cc:194] Registered new tserver with Master: ea6bac8f757445e48100b025a25b24e2 (127.0.197.65:34557)
I20260929 16:42:26.438570 9098 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.65@KRBTEST.COM'} at 127.0.197.65:60081
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab.
Entry for principal kudu/127.0.197.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab.
I20260929 16:42:26.473083 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.66:0
--local_ip_for_outbound_sockets=127.0.197.66
--webserver_interface=127.0.197.66
--webserver_port=0
--tserver_master_addrs=127.0.197.126:41429
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab
--principal=kudu/127.0.197.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:45967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:42:26.827280 9293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:42:26.827605 9293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:42:26.827747 9293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:42:26.835906 9293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:42:26.836043 9293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:42:26.836226 9293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.66
I20260929 16:42:26.846832 9293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:45967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab
--principal=kudu/127.0.197.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.197.66
--webserver_port=0
--tserver_master_addrs=127.0.197.126:41429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.66
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:42:26.848278 9293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:42:26.849653 9293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:42:26.867451 9301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:26.867405 9299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:26.867475 9298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:42:26.871836 9293 server_base.cc:1034] running on GCE node
I20260929 16:42:26.872609 9293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:42:26.873762 9293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:42:26.875092 9293 hybrid_clock.cc:648] HybridClock initialized: now 1790700146874960 us; error 73 us; skew 500 ppm
I20260929 16:42:26.875450 9293 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:42:26 dist-test-slave-l31f krb5kdc[9053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:42:26.878895 9293 init.cc:377] Logged in from keytab as kudu/127.0.197.66@KRBTEST.COM (short username kudu)
I20260929 16:42:26.880834 9293 webserver.cc:460] Webserver started at http://127.0.197.66:42867/ using document root <none> and password file <none>
I20260929 16:42:26.881630 9293 fs_manager.cc:359] Metadata directory not provided
I20260929 16:42:26.881795 9293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:42:26.882122 9293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:42:26.885308 9293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-1/data/instance:
uuid: "58822904244d4eb5a05384376dafd9fe"
format_stamp: "Formatted at 2026-09-29 16:42:26 on dist-test-slave-l31f"
server_key: "1a8066cb6a63960f36d690eb7611e628"
server_key_iv: "38ac4323597cfae973b745913b607c13"
server_key_version: "encryptionkey@0"
I20260929 16:42:26.886242 9293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-1/wal/instance:
uuid: "58822904244d4eb5a05384376dafd9fe"
format_stamp: "Formatted at 2026-09-29 16:42:26 on dist-test-slave-l31f"
server_key: "1a8066cb6a63960f36d690eb7611e628"
server_key_iv: "38ac4323597cfae973b745913b607c13"
server_key_version: "encryptionkey@0"
I20260929 16:42:26.892169 9293 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 16:42:26.896310 9308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:26.897825 9293 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:42:26.898042 9293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-1/wal
uuid: "58822904244d4eb5a05384376dafd9fe"
format_stamp: "Formatted at 2026-09-29 16:42:26 on dist-test-slave-l31f"
server_key: "1a8066cb6a63960f36d690eb7611e628"
server_key_iv: "38ac4323597cfae973b745913b607c13"
server_key_version: "encryptionkey@0"
I20260929 16:42:26.898291 9293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:42:26.914278 9293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:42:26.919188 9293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:42:26.919515 9293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:42:26.920581 9293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:42:26.922037 9293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:42:26.922153 9293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:26.922315 9293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:42:26.922394 9293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:26.957736 9293 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.66:33143
I20260929 16:42:26.957762 9420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.66:33143 every 8 connection(s)
I20260929 16:42:26.959249 9293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
I20260929 16:42:26.968098 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 9293
I20260929 16:42:26.968294 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-1/wal/instance
I20260929 16:42:26.968760 789 external_mini_cluster.cc:1428] Setting key 30aa4ce14049bc251cfcbac15c3bcc02
Sep 29 16:42:26 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.66@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:42:26.984769 9423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:26.962572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:43391 (local address 127.0.197.126:41429)
0929 16:42:26.963195 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:26.963202 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:26.964664 (+ 1462us) server_negotiation.cc:407] Connection header received
0929 16:42:26.966721 (+ 2057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:26.966726 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:26.966805 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:26.966917 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:26.968897 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:26.969817 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:26.971817 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:26.972637 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:26.972811 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:26.977168 (+ 4357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:26.977178 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:26.977186 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:26.977243 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:26.980774 (+ 3531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:26.981967 (+ 1193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:26.981972 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:26.981974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:26.982148 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:26.982732 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:26.982736 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:26.982738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:26.983121 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:26.983290 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:26.984111 (+ 821us) server_negotiation.cc:299] Negotiation successful
0929 16:42:26.984368 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":309,"threads_started":1}
I20260929 16:42:26.985800 9424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:26.963642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:41429 (local address 127.0.197.66:43391)
0929 16:42:26.964463 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:26.964506 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:26.966399 (+ 1893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:26.967299 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:26.967310 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:26.967905 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:26.968666 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:26.968688 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:26.970019 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:26.970023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:26.971649 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:26.971664 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:26.972917 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:26.972923 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:26.973162 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:26.973994 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:26.974044 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:26.976790 (+ 2746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:42:26.980998 (+ 4208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:26.981011 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:26.981046 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:26.981767 (+ 721us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:26.982312 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:26.982317 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:26.982321 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:26.982558 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:26.983263 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:42:26.983272 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:42:26.983698 (+ 426us) client_negotiation.cc:769] Sending connection context
0929 16:42:26.983997 (+ 299us) client_negotiation.cc:240] Negotiation successful
0929 16:42:26.984606 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":305,"threads_started":1}
I20260929 16:42:26.987841 9421 heartbeater.cc:347] Connected to a master server at 127.0.197.126:41429
I20260929 16:42:26.988224 9421 heartbeater.cc:464] Registering TS with master...
I20260929 16:42:26.989109 9421 heartbeater.cc:511] Master 127.0.197.126:41429 requested a full tablet report, sending...
I20260929 16:42:26.990818 9098 ts_manager.cc:194] Registered new tserver with Master: 58822904244d4eb5a05384376dafd9fe (127.0.197.66:33143)
I20260929 16:42:26.991719 9098 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.66@KRBTEST.COM'} at 127.0.197.66:43391
WARNING: no policy specified for kudu/127.0.197.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab.
Entry for principal kudu/127.0.197.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab.
I20260929 16:42:27.028461 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.67:0
--local_ip_for_outbound_sockets=127.0.197.67
--webserver_interface=127.0.197.67
--webserver_port=0
--tserver_master_addrs=127.0.197.126:41429
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab
--principal=kudu/127.0.197.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:45967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:42:27.372546 9427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:42:27.372874 9427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:42:27.372982 9427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:42:27.380586 9427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:42:27.380722 9427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:42:27.380842 9427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.67
I20260929 16:42:27.390872 9427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:45967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab
--principal=kudu/127.0.197.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.197.67
--webserver_port=0
--tserver_master_addrs=127.0.197.126:41429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.67
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:42:27.392195 9427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:42:27.393525 9427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:42:27.409497 9432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:27.409536 9433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:27.410048 9435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:42:27.412196 9427 server_base.cc:1034] running on GCE node
I20260929 16:42:27.412993 9427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:42:27.414104 9427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:42:27.415371 9427 hybrid_clock.cc:648] HybridClock initialized: now 1790700147415329 us; error 46 us; skew 500 ppm
I20260929 16:42:27.415743 9427 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:42:27 dist-test-slave-l31f krb5kdc[9053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:42:27.419143 9427 init.cc:377] Logged in from keytab as kudu/127.0.197.67@KRBTEST.COM (short username kudu)
I20260929 16:42:27.420892 9427 webserver.cc:460] Webserver started at http://127.0.197.67:36795/ using document root <none> and password file <none>
I20260929 16:42:27.421681 9427 fs_manager.cc:359] Metadata directory not provided
I20260929 16:42:27.421833 9427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:42:27.422173 9427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:42:27.425364 9427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-2/data/instance:
uuid: "77be215594d24a1abdd72eca84b8d3a5"
format_stamp: "Formatted at 2026-09-29 16:42:27 on dist-test-slave-l31f"
server_key: "0c2c98bfa119c700e42581ec21989aa5"
server_key_iv: "ec13a07c989b6f953bf68d36469ce4a1"
server_key_version: "encryptionkey@0"
I20260929 16:42:27.426230 9427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-2/wal/instance:
uuid: "77be215594d24a1abdd72eca84b8d3a5"
format_stamp: "Formatted at 2026-09-29 16:42:27 on dist-test-slave-l31f"
server_key: "0c2c98bfa119c700e42581ec21989aa5"
server_key_iv: "ec13a07c989b6f953bf68d36469ce4a1"
server_key_version: "encryptionkey@0"
I20260929 16:42:27.432230 9427 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 16:42:27.436074 9442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:27.437475 9427 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:42:27.437681 9427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-2/wal
uuid: "77be215594d24a1abdd72eca84b8d3a5"
format_stamp: "Formatted at 2026-09-29 16:42:27 on dist-test-slave-l31f"
server_key: "0c2c98bfa119c700e42581ec21989aa5"
server_key_iv: "ec13a07c989b6f953bf68d36469ce4a1"
server_key_version: "encryptionkey@0"
I20260929 16:42:27.437903 9427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:42:27.443746 9287 heartbeater.cc:503] Master 127.0.197.126:41429 was elected leader, sending a full tablet report...
I20260929 16:42:27.457906 9427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:42:27.462879 9427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:42:27.463260 9427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:42:27.464571 9427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:42:27.466116 9427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:42:27.466194 9427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:27.466285 9427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:42:27.466328 9427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:27.501482 9427 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.67:35167
I20260929 16:42:27.501539 9554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.67:35167 every 8 connection(s)
I20260929 16:42:27.503077 9427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
I20260929 16:42:27.512848 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 9427
I20260929 16:42:27.513013 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-2/wal/instance
I20260929 16:42:27.513480 789 external_mini_cluster.cc:1428] Setting key 2606b2958b33ed2ace0fabc60bb2b08f
Sep 29 16:42:27 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.67@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:42:27.528430 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:27.506207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:55615 (local address 127.0.197.126:41429)
0929 16:42:27.506668 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:27.506673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:27.507828 (+ 1155us) server_negotiation.cc:407] Connection header received
0929 16:42:27.509731 (+ 1903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:27.509736 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:27.509831 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:27.509956 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:27.511749 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:27.512642 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:27.515817 (+ 3175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:27.516436 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:27.516564 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:27.520999 (+ 4435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:27.521005 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:27.521009 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:27.521047 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:27.524725 (+ 3678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:27.525656 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:27.525661 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:27.525664 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:27.525801 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:27.526306 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:27.526309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:27.526311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:27.526579 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:27.526698 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:27.527784 (+ 1086us) server_negotiation.cc:299] Negotiation successful
0929 16:42:27.528028 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":236,"threads_started":1}
I20260929 16:42:27.530081 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:27.506829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:41429 (local address 127.0.197.67:55615)
0929 16:42:27.507641 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:27.507690 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:27.509403 (+ 1713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:27.510222 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:27.510231 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:27.510782 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:27.511524 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:27.511544 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:27.512868 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:27.512873 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:27.515570 (+ 2697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:27.515594 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:27.516714 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:27.516723 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:27.517194 (+ 471us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:27.518132 (+ 938us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:27.518163 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:27.520691 (+ 2528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:42:27.524967 (+ 4276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:27.524976 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:27.524992 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:27.525494 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:27.525951 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:27.525955 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:27.525957 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:27.526165 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:27.526850 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:42:27.526858 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:42:27.527345 (+ 487us) client_negotiation.cc:769] Sending connection context
0929 16:42:27.527657 (+ 312us) client_negotiation.cc:240] Negotiation successful
0929 16:42:27.528258 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":369,"threads_started":1}
I20260929 16:42:27.531983 9555 heartbeater.cc:347] Connected to a master server at 127.0.197.126:41429
I20260929 16:42:27.532373 9555 heartbeater.cc:464] Registering TS with master...
I20260929 16:42:27.533147 9555 heartbeater.cc:511] Master 127.0.197.126:41429 requested a full tablet report, sending...
I20260929 16:42:27.534796 9098 ts_manager.cc:194] Registered new tserver with Master: 77be215594d24a1abdd72eca84b8d3a5 (127.0.197.67:35167)
I20260929 16:42:27.535683 9098 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.67@KRBTEST.COM'} at 127.0.197.67:55615
I20260929 16:42:27.542412 789 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.197.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.197.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.197.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/oryx_127.0.197.65.keytab.
Entry for principal oryx/127.0.197.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/oryx_127.0.197.65.keytab.
WARNING: no policy specified for oryx/127.0.197.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.197.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.197.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/oryx_127.0.197.66.keytab.
Entry for principal oryx/127.0.197.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/oryx_127.0.197.66.keytab.
WARNING: no policy specified for oryx/127.0.197.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.197.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.197.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/oryx_127.0.197.67.keytab.
Entry for principal oryx/127.0.197.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/oryx_127.0.197.67.keytab.
I20260929 16:42:27.702855 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 9159
Sep 29 16:42:27 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.65@KRBTEST.COM
I20260929 16:42:27.720932 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:27.704183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47624 (local address 127.0.197.65:34557)
0929 16:42:27.705320 (+ 1137us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:27.705328 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:27.705354 (+ 26us) server_negotiation.cc:407] Connection header received
0929 16:42:27.705411 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:27.705418 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:27.705650 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:27.705797 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:27.707281 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:27.708329 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:27.709621 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:27.710218 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:27.710332 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:27.714765 (+ 4433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:27.714781 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:27.714798 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:27.714838 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:27.717773 (+ 2935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:27.718623 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:27.718632 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:27.718634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:27.718808 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:27.719353 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:27.719357 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:27.719359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:27.719757 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:27.720009 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:27.720277 (+ 268us) server_negotiation.cc:299] Negotiation successful
0929 16:42:27.720480 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":987,"thread_start_us":812,"threads_started":1}
I20260929 16:42:27.722215 9282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:42:27.808403 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 9159
I20260929 16:42:27.827484 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 9293
Sep 29 16:42:27 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.66@KRBTEST.COM
I20260929 16:42:27.845666 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:27.828596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45886 (local address 127.0.197.66:33143)
0929 16:42:27.829102 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:27.829110 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:27.829138 (+ 28us) server_negotiation.cc:407] Connection header received
0929 16:42:27.829200 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:27.829206 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:27.829424 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:27.829612 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:27.830880 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:27.832002 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:27.833499 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:27.834096 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:27.834259 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:27.838957 (+ 4698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:27.838973 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:27.838989 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:27.839109 (+ 120us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:27.842274 (+ 3165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:27.843286 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:27.843298 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:27.843304 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:27.843462 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:27.843881 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:27.843886 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:27.843892 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:27.844329 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:27.844651 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:27.844985 (+ 334us) server_negotiation.cc:299] Negotiation successful
0929 16:42:27.845280 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":252,"threads_started":1}
I20260929 16:42:27.847127 9416 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:42:27.931428 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 9293
I20260929 16:42:27.948392 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 9427
Sep 29 16:42:27 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.67@KRBTEST.COM
I20260929 16:42:27.965862 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:27.949445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58432 (local address 127.0.197.67:35167)
0929 16:42:27.949930 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:27.949938 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:27.949969 (+ 31us) server_negotiation.cc:407] Connection header received
0929 16:42:27.950036 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:27.950043 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:27.950252 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:27.950451 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:27.951894 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:27.952960 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:27.954331 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:27.954989 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:27.955199 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:27.959746 (+ 4547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:27.959759 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:27.959775 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:27.959819 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:27.962763 (+ 2944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:27.963623 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:27.963632 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:27.963634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:27.963768 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:27.964173 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:27.964176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:27.964178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:27.964554 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:27.964873 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:27.965202 (+ 329us) server_negotiation.cc:299] Negotiation successful
0929 16:42:27.965489 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":234,"threads_started":1}
I20260929 16:42:27.967546 9550 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:42:28.053445 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 9427
I20260929 16:42:28.070523 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 9066
I20260929 16:42:28.071981 9128 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:42:28.175815 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 9066
I20260929 16:42:28.198024 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:41429
--webserver_interface=127.0.197.126
--webserver_port=37763
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:45967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.126:41429
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:42:28.542414 9573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:42:28.542755 9573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:42:28.542912 9573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:42:28.550504 9573 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:42:28.550642 9573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:42:28.550734 9573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:42:28.550827 9573 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:42:28.550917 9573 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:42:28.561471 9573 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:45967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.197.126:41429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:41429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.197.126
--webserver_port=37763
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:42:28.562909 9573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:42:28.564319 9573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:42:28.577143 9578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:28.577251 9581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:28.577188 9579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:42:28.583261 9573 server_base.cc:1034] running on GCE node
I20260929 16:42:28.584262 9573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:42:28.585968 9573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:42:28.587184 9573 hybrid_clock.cc:648] HybridClock initialized: now 1790700148587141 us; error 53 us; skew 500 ppm
I20260929 16:42:28.587590 9573 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:42:28 dist-test-slave-l31f krb5kdc[9053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:42:28.590816 9573 init.cc:377] Logged in from keytab as kudu/127.0.197.126@KRBTEST.COM (short username kudu)
I20260929 16:42:28.592708 9573 webserver.cc:460] Webserver started at http://127.0.197.126:37763/ using document root <none> and password file <none>
I20260929 16:42:28.593528 9573 fs_manager.cc:359] Metadata directory not provided
I20260929 16:42:28.593703 9573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:42:28.599861 9573 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:42:28.603109 9588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:28.604722 9573 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:42:28.604957 9573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "4feb71ed104840ebbeb9474b650b5e78"
format_stamp: "Formatted at 2026-09-29 16:42:25 on dist-test-slave-l31f"
server_key: "de3bb815d20f5a5d02cf04eaf3ae1d97"
server_key_iv: "f86ac5e0b15175fd69583bd6839a5861"
server_key_version: "encryptionkey@0"
I20260929 16:42:28.606014 9573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:42:28.622721 9573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:42:28.627501 9573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:42:28.627812 9573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:42:28.648429 9573 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.126:41429
I20260929 16:42:28.648442 9639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.126:41429 every 8 connection(s)
I20260929 16:42:28.650060 9573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:42:28.652747 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 9573
I20260929 16:42:28.653717 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.65:34557
--local_ip_for_outbound_sockets=127.0.197.65
--tserver_master_addrs=127.0.197.126:41429
--webserver_port=42585
--webserver_interface=127.0.197.65
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab
--principal=kudu/127.0.197.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:45967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.197.65
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/oryx_127.0.197.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:42:28.661305 9640 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:42:28.665663 9640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4feb71ed104840ebbeb9474b650b5e78: Bootstrap starting.
I20260929 16:42:28.687955 9640 log.cc:824] T 00000000000000000000000000000000 P 4feb71ed104840ebbeb9474b650b5e78: Log is configured to *not* fsync() on all Append() calls
I20260929 16:42:28.697870 9640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4feb71ed104840ebbeb9474b650b5e78: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:42:28.698508 9640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4feb71ed104840ebbeb9474b650b5e78: Bootstrap complete.
I20260929 16:42:28.708483 9640 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4feb71ed104840ebbeb9474b650b5e78 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4feb71ed104840ebbeb9474b650b5e78" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 41429 } }
I20260929 16:42:28.709410 9640 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4feb71ed104840ebbeb9474b650b5e78 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4feb71ed104840ebbeb9474b650b5e78, State: Initialized, Role: FOLLOWER
I20260929 16:42:28.710435 9640 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4feb71ed104840ebbeb9474b650b5e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4feb71ed104840ebbeb9474b650b5e78" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 41429 } }
I20260929 16:42:28.710711 9640 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4feb71ed104840ebbeb9474b650b5e78 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:42:28.710834 9640 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4feb71ed104840ebbeb9474b650b5e78 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:42:28.711081 9640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4feb71ed104840ebbeb9474b650b5e78 [term 1 FOLLOWER]: Advancing to term 2
I20260929 16:42:28.715679 9640 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4feb71ed104840ebbeb9474b650b5e78 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4feb71ed104840ebbeb9474b650b5e78" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 41429 } }
I20260929 16:42:28.716298 9640 leader_election.cc:302] T 00000000000000000000000000000000 P 4feb71ed104840ebbeb9474b650b5e78 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4feb71ed104840ebbeb9474b650b5e78; no voters:
I20260929 16:42:28.716996 9640 leader_election.cc:288] T 00000000000000000000000000000000 P 4feb71ed104840ebbeb9474b650b5e78 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 16:42:28.717149 9644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4feb71ed104840ebbeb9474b650b5e78 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 16:42:28.718459 9644 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4feb71ed104840ebbeb9474b650b5e78 [term 2 LEADER]: Becoming Leader. State: Replica: 4feb71ed104840ebbeb9474b650b5e78, State: Running, Role: LEADER
I20260929 16:42:28.719143 9644 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4feb71ed104840ebbeb9474b650b5e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4feb71ed104840ebbeb9474b650b5e78" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 41429 } }
I20260929 16:42:28.719550 9640 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4feb71ed104840ebbeb9474b650b5e78 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:42:28.721999 9645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4feb71ed104840ebbeb9474b650b5e78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4feb71ed104840ebbeb9474b650b5e78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4feb71ed104840ebbeb9474b650b5e78" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 41429 } } }
I20260929 16:42:28.722492 9645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4feb71ed104840ebbeb9474b650b5e78 [sys.catalog]: This master's current role is: LEADER
I20260929 16:42:28.722810 9646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4feb71ed104840ebbeb9474b650b5e78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4feb71ed104840ebbeb9474b650b5e78. Latest consensus state: current_term: 2 leader_uuid: "4feb71ed104840ebbeb9474b650b5e78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4feb71ed104840ebbeb9474b650b5e78" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 41429 } } }
I20260929 16:42:28.723301 9646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4feb71ed104840ebbeb9474b650b5e78 [sys.catalog]: This master's current role is: LEADER
I20260929 16:42:28.731588 9649 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:42:28.740207 9649 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:42:28.742347 9649 catalog_manager.cc:1259] Loaded cluster ID: 8b6884fa10a746faae2b91e482af1614
I20260929 16:42:28.742512 9649 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:42:28.748333 9649 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:42:28.750651 9649 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4feb71ed104840ebbeb9474b650b5e78: Loaded TSK: 0
I20260929 16:42:28.752900 9649 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 16:42:29.057277 9642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:42:29.057677 9642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:42:29.057843 9642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:42:29.067441 9642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:42:29.067654 9642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:42:29.067863 9642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.65
I20260929 16:42:29.079272 9642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:45967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/oryx_127.0.197.65.keytab
--principal=oryx/127.0.197.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.65:34557
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.197.65
--webserver_port=42585
--tserver_master_addrs=127.0.197.126:41429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.65
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:42:29.080848 9642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:42:29.082247 9642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:42:29.097191 9667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:29.097723 9666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:29.098158 9669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:42:29.098415 9642 server_base.cc:1034] running on GCE node
I20260929 16:42:29.099287 9642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:42:29.100446 9642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:42:29.101735 9642 hybrid_clock.cc:648] HybridClock initialized: now 1790700149101682 us; error 48 us; skew 500 ppm
I20260929 16:42:29.102171 9642 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:42:29 dist-test-slave-l31f krb5kdc[9053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700149, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.197.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:42:29.105607 9642 init.cc:377] Logged in from keytab as oryx/127.0.197.65@KRBTEST.COM (short username oryx)
I20260929 16:42:29.107476 9642 webserver.cc:460] Webserver started at http://127.0.197.65:42585/ using document root <none> and password file <none>
I20260929 16:42:29.108273 9642 fs_manager.cc:359] Metadata directory not provided
I20260929 16:42:29.108426 9642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:42:29.114729 9642 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260929 16:42:29.117877 9676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:29.119643 9642 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:42:29.119844 9642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-0/wal
uuid: "ea6bac8f757445e48100b025a25b24e2"
format_stamp: "Formatted at 2026-09-29 16:42:26 on dist-test-slave-l31f"
server_key: "95b7938121eb01788eda5b0a1cbf2808"
server_key_iv: "06a6c2b703564a49f64c6845513eb2cf"
server_key_version: "encryptionkey@0"
I20260929 16:42:29.120990 9642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:42:29.135352 9642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:42:29.140141 9642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:42:29.140445 9642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:42:29.141513 9642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:42:29.143072 9642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:42:29.143181 9642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:29.143337 9642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:42:29.143420 9642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:29.179457 9642 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.65:34557
I20260929 16:42:29.179522 9788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.65:34557 every 8 connection(s)
I20260929 16:42:29.180981 9642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
I20260929 16:42:29.184509 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 9642
I20260929 16:42:29.185261 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.66:33143
--local_ip_for_outbound_sockets=127.0.197.66
--tserver_master_addrs=127.0.197.126:41429
--webserver_port=42867
--webserver_interface=127.0.197.66
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab
--principal=kudu/127.0.197.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:45967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.197.66
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/oryx_127.0.197.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:42:29 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.65@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:42:29 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.65@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
W20260929 16:42:29.543534 9793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:42:29.543848 9793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:42:29.544014 9793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:42:29.551924 9793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:42:29.552053 9793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:42:29.552249 9793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.66
I20260929 16:42:29.562614 9793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:45967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/oryx_127.0.197.66.keytab
--principal=oryx/127.0.197.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.66:33143
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.197.66
--webserver_port=42867
--tserver_master_addrs=127.0.197.126:41429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.66
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:42:29.564095 9793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:42:29.565408 9793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:42:29.580905 9799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:29.580878 9802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:29.581035 9800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:42:29.583122 9793 server_base.cc:1034] running on GCE node
I20260929 16:42:29.583875 9793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:42:29.585023 9793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:42:29.586303 9793 hybrid_clock.cc:648] HybridClock initialized: now 1790700149586232 us; error 59 us; skew 500 ppm
I20260929 16:42:29.586738 9793 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:42:29 dist-test-slave-l31f krb5kdc[9053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700149, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.197.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:42:29.590353 9793 init.cc:377] Logged in from keytab as oryx/127.0.197.66@KRBTEST.COM (short username oryx)
I20260929 16:42:29.592247 9793 webserver.cc:460] Webserver started at http://127.0.197.66:42867/ using document root <none> and password file <none>
I20260929 16:42:29.593017 9793 fs_manager.cc:359] Metadata directory not provided
I20260929 16:42:29.593138 9793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:42:29.599216 9793 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260929 16:42:29.602335 9809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:29.604440 9793 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260929 16:42:29.604671 9793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-1/wal
uuid: "58822904244d4eb5a05384376dafd9fe"
format_stamp: "Formatted at 2026-09-29 16:42:26 on dist-test-slave-l31f"
server_key: "1a8066cb6a63960f36d690eb7611e628"
server_key_iv: "38ac4323597cfae973b745913b607c13"
server_key_version: "encryptionkey@0"
I20260929 16:42:29.605715 9793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:42:29.633039 9793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:42:29.637984 9793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:42:29.638289 9793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:42:29.639832 9793 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:42:29.641420 9793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:42:29.641551 9793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:29.641690 9793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:42:29.641782 9793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:29.688752 9793 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.66:33143
I20260929 16:42:29.688804 9921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.66:33143 every 8 connection(s)
I20260929 16:42:29.690286 9793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
I20260929 16:42:29.700978 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 9793
I20260929 16:42:29.702080 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.67:35167
--local_ip_for_outbound_sockets=127.0.197.67
--tserver_master_addrs=127.0.197.126:41429
--webserver_port=36795
--webserver_interface=127.0.197.67
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab
--principal=kudu/127.0.197.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:45967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.197.67
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/oryx_127.0.197.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:42:29 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.66@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:42:29 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.66@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
W20260929 16:42:30.079877 9926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:42:30.080360 9926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:42:30.080607 9926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:42:30.088516 9926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:42:30.088716 9926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:42:30.088917 9926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.67
I20260929 16:42:30.100183 9926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:45967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/krb5kdc/oryx_127.0.197.67.keytab
--principal=oryx/127.0.197.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.67:35167
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.197.67
--webserver_port=36795
--tserver_master_addrs=127.0.197.126:41429
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.67
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:42:30.101788 9926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:42:30.103334 9926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:42:30.123114 9933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:30.131560 9932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:30.133637 9936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:42:30.134193 9926 server_base.cc:1034] running on GCE node
I20260929 16:42:30.135474 9926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:42:30.137383 9926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:42:30.140905 9926 hybrid_clock.cc:648] HybridClock initialized: now 1790700150140785 us; error 91 us; skew 500 ppm
I20260929 16:42:30.141629 9926 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:42:30 dist-test-slave-l31f krb5kdc[9053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700150, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.197.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:42:30.148777 9926 init.cc:377] Logged in from keytab as oryx/127.0.197.67@KRBTEST.COM (short username oryx)
I20260929 16:42:30.152233 9926 webserver.cc:460] Webserver started at http://127.0.197.67:36795/ using document root <none> and password file <none>
I20260929 16:42:30.153800 9926 fs_manager.cc:359] Metadata directory not provided
I20260929 16:42:30.154094 9926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:42:30.166795 9926 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.009s sys 0.000s
I20260929 16:42:30.173357 9942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:30.176584 9926 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.004s sys 0.002s
I20260929 16:42:30.177065 9926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-2/wal
uuid: "77be215594d24a1abdd72eca84b8d3a5"
format_stamp: "Formatted at 2026-09-29 16:42:27 on dist-test-slave-l31f"
server_key: "0c2c98bfa119c700e42581ec21989aa5"
server_key_iv: "ec13a07c989b6f953bf68d36469ce4a1"
server_key_version: "encryptionkey@0"
I20260929 16:42:30.179705 9926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:42:30.224382 9926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:42:30.233606 9926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:42:30.234083 9926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:42:30.236047 9926 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:42:30.238963 9926 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:42:30.239195 9926 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260929 16:42:30.239384 9926 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:42:30.239501 9926 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:30.305819 9926 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.67:35167
I20260929 16:42:30.305821 10054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.67:35167 every 8 connection(s)
I20260929 16:42:30.309150 9926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
I20260929 16:42:30.319273 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 9926
Sep 29 16:42:30 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.67@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:42:30 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.67@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:42:30.383622 10057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:30.320874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53880 (local address 127.0.197.126:41429)
0929 16:42:30.348157 (+ 27283us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:30.348166 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:30.348193 (+ 27us) server_negotiation.cc:407] Connection header received
0929 16:42:30.348267 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:30.348275 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:30.348374 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:30.348540 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:30.352131 (+ 3591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:30.354500 (+ 2369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:30.357743 (+ 3243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:30.358824 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:30.359050 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:30.366482 (+ 7432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:30.366502 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:30.366542 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:30.366647 (+ 105us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:30.374000 (+ 7353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:30.375721 (+ 1721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:30.375738 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:30.375756 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:30.376136 (+ 380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:30.376917 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:30.376926 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:30.376932 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:30.378002 (+ 1070us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:30.380384 (+ 2382us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:30.380867 (+ 483us) server_negotiation.cc:299] Negotiation successful
0929 16:42:30.381742 (+ 875us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":105,"server-negotiator.queue_time_us":27059,"thread_start_us":15532,"threads_started":1}
W20260929 16:42:30.671340 9792 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.476s user 0.001s sys 0.014s
W20260929 16:42:30.673552 9791 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:42:29.184126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:56637 (local address 127.0.197.126:41429)
0929 16:42:29.185016 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:29.185043 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:29.186003 (+ 960us) server_negotiation.cc:407] Connection header received
0929 16:42:29.188022 (+ 2019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:29.188035 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:29.188416 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:29.188827 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:29.190490 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:29.191844 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:29.193504 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:29.194191 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:29.194412 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:30.673030 (+1478618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.197.65:56637: BlockingRecv error: recv got EOF from 127.0.197.65:56637 (error 108)
Metrics: {"server-negotiator.queue_time_us":554,"spinlock_wait_cycles":18304,"thread_start_us":434,"threads_started":1}
I20260929 16:42:30.675304 9792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:29.184233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:41429 (local address 127.0.197.65:56637)
0929 16:42:29.185482 (+ 1249us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:29.185535 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:29.187402 (+ 1867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:29.188985 (+ 1583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:29.188996 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:29.189657 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:29.190242 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:29.190261 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:29.192054 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:29.192058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:29.193315 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:29.193328 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:29.194450 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:29.194457 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:29.194747 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:29.195545 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:29.195570 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:30.672479 (+1476909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.197.126:41429: Server oryx/127.0.197.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":970,"thread_start_us":808,"threads_started":1}
W20260929 16:42:30.678011 9789 heartbeater.cc:644] Failed to heartbeat to 127.0.197.126:41429 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.197.126:41429: Client connection negotiation failed: client connection to 127.0.197.126:41429: Server oryx/127.0.197.126@KRBTEST.COM not found in Kerberos database
Sep 29 16:42:30 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.65@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:42:30 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.65@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
W20260929 16:42:30.709754 9791 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:42:30.679329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:33881 (local address 127.0.197.126:41429)
0929 16:42:30.679616 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:30.679626 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:30.679767 (+ 141us) server_negotiation.cc:407] Connection header received
0929 16:42:30.680015 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:30.680024 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:30.680144 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:30.680318 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:30.681775 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:30.683627 (+ 1852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:30.686847 (+ 3220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:30.687995 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:30.688180 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:30.709331 (+ 21151us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.197.65:33881: BlockingRecv error: recv got EOF from 127.0.197.65:33881 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 16:42:30.709692 9792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:30.679205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:41429 (local address 127.0.197.65:33881)
0929 16:42:30.679622 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:30.679650 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:30.679839 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:30.680389 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:30.680397 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:30.681181 (+ 784us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:30.681523 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:30.681542 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:30.683901 (+ 2359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:30.683912 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:30.686512 (+ 2600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:30.686546 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:30.688574 (+ 2028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:30.688605 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:30.688915 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:30.700019 (+ 11104us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:30.700064 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:30.708884 (+ 8820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.197.126:41429: Server oryx/127.0.197.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225}
Sep 29 16:42:30 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.65@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:42:30 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.65@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:42:30.729517 9792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:30.711447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:41429 (local address 127.0.197.65:38251)
0929 16:42:30.711711 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:30.711744 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:30.712243 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:30.712875 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:30.712883 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:30.713553 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:30.713930 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:30.713948 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:30.716353 (+ 2405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:30.716364 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:30.718842 (+ 2478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:30.718868 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:30.720910 (+ 2042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:30.720921 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:30.721251 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:30.722323 (+ 1072us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:30.722364 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:30.729042 (+ 6678us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.197.126:41429: Server oryx/127.0.197.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":85}
W20260929 16:42:30.729905 9791 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:42:30.712004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:38251 (local address 127.0.197.126:41429)
0929 16:42:30.712273 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:30.712283 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:30.712318 (+ 35us) server_negotiation.cc:407] Connection header received
0929 16:42:30.712444 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:30.712454 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:30.712571 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:30.712792 (+ 221us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:30.714236 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:30.716060 (+ 1824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:30.719174 (+ 3114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:30.720306 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:30.720461 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:30.729480 (+ 9019us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.197.65:38251: BlockingRecv error: recv got EOF from 127.0.197.65:38251 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260929 16:42:30.730363 9789 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 16:42:31.053853 9925 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.349s user 0.000s sys 0.015s
W20260929 16:42:31.055244 9924 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:42:29.693455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:35123 (local address 127.0.197.126:41429)
0929 16:42:29.693934 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:29.693939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:29.695820 (+ 1881us) server_negotiation.cc:407] Connection header received
0929 16:42:29.697840 (+ 2020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:29.697846 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:29.697948 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:29.698086 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:29.699791 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:29.700578 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:29.702436 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:29.702908 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:29.703082 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:31.054880 (+1351798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.197.66:35123: BlockingRecv error: recv got EOF from 127.0.197.66:35123 (error 108)
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":263,"threads_started":1}
I20260929 16:42:31.055918 9925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:29.694658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:41429 (local address 127.0.197.66:35123)
0929 16:42:29.695644 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:29.695687 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:29.697520 (+ 1833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:29.698357 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:29.698367 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:29.698909 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:29.699586 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:29.699605 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:29.700773 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:29.700778 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:29.702195 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:29.702238 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:29.703202 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:29.703213 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:29.703486 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:29.704261 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:29.704287 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:31.054689 (+1350402us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.197.126:41429: Server oryx/127.0.197.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":687,"thread_start_us":424,"threads_started":1}
W20260929 16:42:31.057197 9922 heartbeater.cc:644] Failed to heartbeat to 127.0.197.126:41429 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.197.126:41429: Client connection negotiation failed: client connection to 127.0.197.126:41429: Server oryx/127.0.197.126@KRBTEST.COM not found in Kerberos database
Sep 29 16:42:31 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.66@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:42:31 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.66@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:42:31.068818 9925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:31.058101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:41429 (local address 127.0.197.66:51635)
0929 16:42:31.058395 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:31.058424 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:31.058633 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:31.059041 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:31.059049 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:31.059758 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:31.060095 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:31.060112 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:31.061309 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:31.061314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:31.062524 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:31.062542 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:31.064033 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:31.064038 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:31.064180 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:31.064985 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:31.065002 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:31.068594 (+ 3592us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.197.126:41429: Server oryx/127.0.197.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":62}
W20260929 16:42:31.070096 9924 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:42:31.058256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:51635 (local address 127.0.197.126:41429)
0929 16:42:31.058420 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:31.058424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:31.058522 (+ 98us) server_negotiation.cc:407] Connection header received
0929 16:42:31.058775 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:31.058778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:31.058840 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:31.058922 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:31.060313 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:31.061129 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:31.062710 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:31.063816 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:31.064012 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:31.069419 (+ 5407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.197.66:51635: BlockingRecv error: recv got EOF from 127.0.197.66:51635 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 29 16:42:31 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.66@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:42:31 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.66@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:42:31.082841 9925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:31.069655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:41429 (local address 127.0.197.66:34029)
0929 16:42:31.069887 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:31.069902 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:31.070020 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:31.070509 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:31.070518 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:31.071179 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:31.071501 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:31.071521 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:31.073104 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:31.073114 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:31.075579 (+ 2465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:31.075618 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:31.076456 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:31.076467 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:31.076797 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:31.077650 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:31.077688 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:31.082530 (+ 4842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.197.126:41429: Server oryx/127.0.197.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147}
W20260929 16:42:31.083072 9791 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:42:31.069902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:34029 (local address 127.0.197.126:41429)
0929 16:42:31.070114 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:31.070120 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:31.070136 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:42:31.070218 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:31.070226 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:31.070300 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:31.070394 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:31.071769 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:31.072768 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:31.075796 (+ 3028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:31.076244 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:31.076347 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:31.082796 (+ 6449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.197.66:34029: BlockingRecv error: recv got EOF from 127.0.197.66:34029 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260929 16:42:31.083362 9922 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 16:42:31.583838 10058 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.221s user 0.008s sys 0.018s
W20260929 16:42:31.585211 10060 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:42:30.315543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:49059 (local address 127.0.197.126:41429)
0929 16:42:30.336389 (+ 20846us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:30.336401 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:30.346732 (+ 10331us) server_negotiation.cc:407] Connection header received
0929 16:42:30.346854 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:30.346863 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:30.347081 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:30.347239 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:30.352087 (+ 4848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:30.354499 (+ 2412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:30.358884 (+ 4385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:30.360056 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:30.360273 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:31.584929 (+1224656us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.197.67:49059: BlockingRecv error: recv got EOF from 127.0.197.67:49059 (error 108)
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":20615,"thread_start_us":32194,"threads_started":1}
I20260929 16:42:31.585839 10058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:30.317410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:41429 (local address 127.0.197.67:49059)
0929 16:42:30.341393 (+ 23983us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:30.341499 (+ 106us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:30.345959 (+ 4460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:30.348663 (+ 2704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:30.348689 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:30.350075 (+ 1386us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:30.351545 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:30.351594 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:30.354897 (+ 3303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:30.354908 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:30.358512 (+ 3604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:30.358549 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:30.360828 (+ 2279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:30.360839 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:30.361379 (+ 540us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:30.362557 (+ 1178us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:30.362621 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:31.584769 (+1222148us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.197.126:41429: Server oryx/127.0.197.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":23426,"thread_start_us":23047,"threads_started":1}
W20260929 16:42:31.586764 10055 heartbeater.cc:644] Failed to heartbeat to 127.0.197.126:41429 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.197.126:41429: Client connection negotiation failed: client connection to 127.0.197.126:41429: Server oryx/127.0.197.126@KRBTEST.COM not found in Kerberos database
Sep 29 16:42:31 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.67@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:42:31 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.67@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:42:31.598037 10058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:31.587494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:41429 (local address 127.0.197.67:39445)
0929 16:42:31.587731 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:31.587748 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:31.588084 (+ 336us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:31.588441 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:31.588445 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:31.588822 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:31.588990 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:31.588998 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:31.590128 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:31.590133 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:31.591368 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:31.591382 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:31.592147 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:31.592152 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:31.592276 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:31.593013 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:31.593031 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:31.597813 (+ 4782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.197.126:41429: Server oryx/127.0.197.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99}
W20260929 16:42:31.598242 10060 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:42:31.587606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:39445 (local address 127.0.197.126:41429)
0929 16:42:31.587774 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:31.587778 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:31.587796 (+ 18us) server_negotiation.cc:407] Connection header received
0929 16:42:31.588242 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:31.588246 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:31.588308 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:31.588395 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:31.589162 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:31.589930 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:31.591531 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:31.591959 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:31.592066 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:31.598014 (+ 5948us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.197.67:39445: BlockingRecv error: recv got EOF from 127.0.197.67:39445 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 29 16:42:31 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.67@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:42:31 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.67@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:42:31.608372 10058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:31.599276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:41429 (local address 127.0.197.67:50897)
0929 16:42:31.599457 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:31.599478 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:31.599594 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:31.599888 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:31.599891 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:31.600194 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:31.600371 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:31.600380 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:31.601411 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:31.601415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:31.602365 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:31.602377 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:31.603285 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:31.603290 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:31.603445 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:31.604167 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:31.604185 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:31.608160 (+ 3975us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.197.126:41429: Server oryx/127.0.197.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 16:42:31.608747 10055 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 16:42:31.608983 10060 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:42:31.599376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:50897 (local address 127.0.197.126:41429)
0929 16:42:31.599565 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:31.599569 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:31.599586 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:42:31.599703 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:31.599707 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:31.599763 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:31.599840 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:31.600527 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:31.601271 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:31.602519 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:31.602987 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:31.603187 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:31.608717 (+ 5530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.197.67:50897: BlockingRecv error: recv got EOF from 127.0.197.67:50897 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Sep 29 16:42:31 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.65@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:42:31 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.65@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:42:31.742120 10062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:31.731243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:41429 (local address 127.0.197.65:38181)
0929 16:42:31.731977 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:31.732004 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:31.732155 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:31.732651 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:31.732659 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:31.733150 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:31.733316 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:31.733327 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:31.734668 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:31.734672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:31.736146 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:31.736180 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:31.737171 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:31.737181 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:31.737381 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:31.738022 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:31.738039 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:31.741828 (+ 3789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.197.126:41429: Server oryx/127.0.197.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":473,"threads_started":1}
W20260929 16:42:31.742411 10060 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:42:31.731622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:38181 (local address 127.0.197.126:41429)
0929 16:42:31.731850 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:31.731857 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:31.732153 (+ 296us) server_negotiation.cc:407] Connection header received
0929 16:42:31.732283 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:31.732287 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:31.732389 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:31.732508 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:31.733533 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:31.734477 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:31.736479 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:31.736932 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:31.737088 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:31.742121 (+ 5033us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.197.65:38181: BlockingRecv error: recv got EOF from 127.0.197.65:38181 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 29 16:42:32 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.66@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:42:32 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.66@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:42:32.093784 10063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:32.084019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:41429 (local address 127.0.197.66:43887)
0929 16:42:32.084563 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:32.084581 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:32.084702 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:32.085009 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:32.085025 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:32.085433 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:32.085585 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:32.085594 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:32.086639 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:32.086643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:32.087624 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:32.087636 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:32.088390 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:32.088395 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:32.088518 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:32.089210 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:32.089246 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:32.093412 (+ 4166us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.197.126:41429: Server oryx/127.0.197.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":337,"threads_started":1}
W20260929 16:42:32.093880 10060 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:42:32.084231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:43887 (local address 127.0.197.126:41429)
0929 16:42:32.084496 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:32.084502 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:32.084661 (+ 159us) server_negotiation.cc:407] Connection header received
0929 16:42:32.084804 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:32.084808 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:32.084871 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:32.084960 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:32.085736 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:32.086495 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:32.087761 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:32.088213 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:32.088318 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:32.093635 (+ 5317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.197.66:43887: BlockingRecv error: recv got EOF from 127.0.197.66:43887 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
Sep 29 16:42:32 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.67@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:42:32 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.67@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:42:32.619822 10064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:32.609555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:41429 (local address 127.0.197.67:37003)
0929 16:42:32.610185 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:32.610207 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:32.610405 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:32.610821 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:32.610824 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:32.611548 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:32.611786 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:32.611797 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:32.612866 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:32.612871 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:32.613855 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:32.613867 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:32.614749 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:32.614754 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:32.614882 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:32.615515 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:32.615534 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:32.619569 (+ 4035us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.197.126:41429: Server oryx/127.0.197.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":342,"threads_started":1}
W20260929 16:42:32.620095 10065 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:42:32.609633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:37003 (local address 127.0.197.126:41429)
0929 16:42:32.610444 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:32.610450 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:32.610469 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:42:32.610565 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:32.610568 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:32.610668 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:32.610797 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:32.611927 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:32.612672 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:32.614027 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:32.614547 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:32.614687 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:32.619828 (+ 5141us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.197.67:37003: BlockingRecv error: recv got EOF from 127.0.197.67:37003 (error 108)
Metrics: {"server-negotiator.queue_time_us":706,"thread_start_us":608,"threads_started":1}
Sep 29 16:42:32 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.65@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:42:32 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.65@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:42:32.753127 10066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:32.743397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:41429 (local address 127.0.197.65:37903)
0929 16:42:32.743897 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:32.743914 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:32.744036 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:32.744407 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:32.744410 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:32.744782 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:32.744947 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:32.744954 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:32.746052 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:32.746056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:32.747125 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:32.747137 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:32.747927 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:32.747933 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:32.748116 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:32.748863 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:32.748887 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:32.752856 (+ 3969us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.197.126:41429: Server oryx/127.0.197.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":308,"threads_started":1}
W20260929 16:42:32.753348 10065 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:42:32.743466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:37903 (local address 127.0.197.126:41429)
0929 16:42:32.743734 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:32.743742 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:32.744001 (+ 259us) server_negotiation.cc:407] Connection header received
0929 16:42:32.744189 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:32.744194 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:32.744268 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:32.744362 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:32.745106 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:32.745911 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:32.747279 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:32.747722 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:32.747900 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:32.753116 (+ 5216us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.197.65:37903: BlockingRecv error: recv got EOF from 127.0.197.65:37903 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 29 16:42:33 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.66@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:42:33 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.66@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:42:33.105545 10067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:33.095289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:41429 (local address 127.0.197.66:53797)
0929 16:42:33.095985 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:33.096002 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:33.096195 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:33.096558 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:33.096563 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:33.097019 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:33.097206 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:33.097216 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:33.098280 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:33.098287 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:33.099273 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:33.099287 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:33.100214 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:33.100220 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:33.100455 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:33.101255 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:33.101274 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:33.105276 (+ 4002us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.197.126:41429: Server oryx/127.0.197.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":471,"threads_started":1}
W20260929 16:42:33.105803 10065 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:42:33.095399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:53797 (local address 127.0.197.126:41429)
0929 16:42:33.095764 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:33.095774 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:33.096109 (+ 335us) server_negotiation.cc:407] Connection header received
0929 16:42:33.096333 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:33.096338 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:33.096411 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:33.096525 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:33.097384 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:33.098125 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:33.099473 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:33.099984 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:33.100123 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:33.105522 (+ 5399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.197.66:53797: BlockingRecv error: recv got EOF from 127.0.197.66:53797 (error 108)
Metrics: {"server-negotiator.queue_time_us":208}
Sep 29 16:42:33 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.67@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:42:33 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.67@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:42:33.630537 10068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:33.620883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:41429 (local address 127.0.197.67:54859)
0929 16:42:33.621545 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:33.621567 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:33.621744 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:33.622149 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:33.622154 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:33.622587 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:33.622843 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:33.622865 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:33.623935 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:33.623941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:33.624949 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:33.624961 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:33.625705 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:33.625711 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:33.625900 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:33.626545 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:33.626564 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:33.630166 (+ 3602us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.197.126:41429: Server oryx/127.0.197.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":312,"threads_started":1}
W20260929 16:42:33.630686 10069 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:42:33.621078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:54859 (local address 127.0.197.126:41429)
0929 16:42:33.621739 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:33.621743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:33.621759 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:42:33.621880 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:33.621887 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:33.621983 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:33.622087 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:33.622991 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:33.623784 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:33.625103 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:33.625535 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:33.625674 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:33.630409 (+ 4735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.197.67:54859: BlockingRecv error: recv got EOF from 127.0.197.67:54859 (error 108)
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":378,"threads_started":1}
Sep 29 16:42:33 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.65@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:42:33 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.65@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:42:33.764959 10070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:33.754463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:41429 (local address 127.0.197.65:37979)
0929 16:42:33.754970 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:33.754988 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:33.755171 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:33.755592 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:33.755596 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:33.755978 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:33.756168 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:33.756177 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:33.757307 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:33.757312 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:33.758235 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:33.758246 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:33.759423 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:33.759430 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:33.759620 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:33.760211 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:33.760228 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:33.764653 (+ 4425us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.197.126:41429: Server oryx/127.0.197.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":272,"threads_started":1}
W20260929 16:42:33.765093 10069 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:42:33.754560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:37979 (local address 127.0.197.126:41429)
0929 16:42:33.755095 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:33.755102 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:33.755139 (+ 37us) server_negotiation.cc:407] Connection header received
0929 16:42:33.755347 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:33.755352 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:33.755420 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:33.755516 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:33.756369 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:33.757179 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:33.758406 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:33.759236 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:33.759402 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:33.764886 (+ 5484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.197.65:37979: BlockingRecv error: recv got EOF from 127.0.197.65:37979 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 29 16:42:34 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.66@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:42:34 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.66@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:42:34.116837 10071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:34.106537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:41429 (local address 127.0.197.66:43041)
0929 16:42:34.107291 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:34.107318 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:34.107436 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:34.107777 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:34.107780 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:34.108130 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:34.108297 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:34.108306 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:34.109391 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:34.109397 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:34.110449 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:34.110464 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:34.111236 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:34.111243 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:34.111430 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:34.112092 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:34.112111 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:34.116549 (+ 4438us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.197.126:41429: Server oryx/127.0.197.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":505,"threads_started":1}
W20260929 16:42:34.117136 10069 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:42:34.106581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:43041 (local address 127.0.197.126:41429)
0929 16:42:34.106857 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:34.106867 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:34.107418 (+ 551us) server_negotiation.cc:407] Connection header received
0929 16:42:34.107536 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:34.107541 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:34.107649 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:34.107735 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:34.108431 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:34.109183 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:34.110593 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:34.111069 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:34.111218 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:34.116844 (+ 5626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.197.66:43041: BlockingRecv error: recv got EOF from 127.0.197.66:43041 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 29 16:42:34 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.67@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:42:34 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.67@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:42:34.641716 10072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:34.631867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:41429 (local address 127.0.197.67:36479)
0929 16:42:34.632440 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:34.632458 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:34.632602 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:34.633111 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:34.633122 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:34.633477 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:34.633645 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:34.633654 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:34.634766 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:34.634770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:34.635716 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:34.635729 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:34.636540 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:34.636546 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:34.636725 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:34.637455 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:34.637479 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:34.641301 (+ 3822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.197.126:41429: Server oryx/127.0.197.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":303,"threads_started":1}
W20260929 16:42:34.641957 10073 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:42:34.631971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:36479 (local address 127.0.197.126:41429)
0929 16:42:34.632537 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:34.632543 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:34.632565 (+ 22us) server_negotiation.cc:407] Connection header received
0929 16:42:34.632838 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:34.632843 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:34.632927 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:34.633036 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:34.633817 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:34.634550 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:34.635890 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:34.636388 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:34.636495 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:34.641575 (+ 5080us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.197.67:36479: BlockingRecv error: recv got EOF from 127.0.197.67:36479 (error 108)
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":304,"threads_started":1}
Sep 29 16:42:34 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.65@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:42:34 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.65@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:42:34.775516 10074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:34.766192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:41429 (local address 127.0.197.65:42301)
0929 16:42:34.766756 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:34.766776 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:34.766908 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:34.767290 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:34.767295 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:34.767705 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:34.767862 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:34.767870 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:34.768874 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:34.768881 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:34.769908 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:34.769920 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:34.770667 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:34.770673 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:34.770869 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:34.771637 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:34.771661 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:34.775251 (+ 3590us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.197.126:41429: Server oryx/127.0.197.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":284,"threads_started":1}
W20260929 16:42:34.775753 10073 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:42:34.766275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:42301 (local address 127.0.197.126:41429)
0929 16:42:34.766524 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:34.766530 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:34.766891 (+ 361us) server_negotiation.cc:407] Connection header received
0929 16:42:34.767076 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:34.767081 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:34.767150 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:34.767260 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:34.768009 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:34.768731 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:34.770075 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:34.770497 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:34.770637 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:34.775485 (+ 4848us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.197.65:42301: BlockingRecv error: recv got EOF from 127.0.197.65:42301 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 29 16:42:35 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.66@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:42:35 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.66@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:42:35.128571 10075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:35.118148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:41429 (local address 127.0.197.66:39427)
0929 16:42:35.118701 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:35.118718 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:35.118868 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:35.119353 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:35.119357 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:35.119782 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:35.119949 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:35.119959 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:35.121163 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:35.121170 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:35.122170 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:35.122184 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:35.122941 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:35.122947 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:35.123168 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:35.123815 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:35.123833 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:35.128279 (+ 4446us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.197.126:41429: Server oryx/127.0.197.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":317,"threads_started":1}
W20260929 16:42:35.128818 10073 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:42:35.118230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:39427 (local address 127.0.197.126:41429)
0929 16:42:35.118494 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:35.118502 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:35.118848 (+ 346us) server_negotiation.cc:407] Connection header received
0929 16:42:35.118977 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:35.118981 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:35.119153 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:35.119273 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:35.120139 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:35.121012 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:35.122339 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:35.122762 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:35.122921 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:35.128536 (+ 5615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.197.66:39427: BlockingRecv error: recv got EOF from 127.0.197.66:39427 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 29 16:42:35 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.67@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:42:35 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.67@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:42:35.662883 10076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:35.643667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:41429 (local address 127.0.197.67:44161)
0929 16:42:35.644549 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:35.644570 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:35.644738 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:35.645442 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:35.645450 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:35.645978 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:35.646214 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:35.646279 (+ 65us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:35.648215 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:35.648223 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:35.650041 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:35.650067 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:35.651389 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:35.651411 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:35.651766 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:35.658164 (+ 6398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:35.658189 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:35.662511 (+ 4322us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.197.126:41429: Server oryx/127.0.197.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":718,"thread_start_us":410,"threads_started":1}
W20260929 16:42:35.663077 10077 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:42:35.643889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:44161 (local address 127.0.197.126:41429)
0929 16:42:35.644445 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:35.644450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:35.644739 (+ 289us) server_negotiation.cc:407] Connection header received
0929 16:42:35.645010 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:35.645017 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:35.645131 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:35.645310 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:35.646605 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:35.647941 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:35.650278 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:35.650919 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:35.651118 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:35.662758 (+ 11640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.197.67:44161: BlockingRecv error: recv got EOF from 127.0.197.67:44161 (error 108)
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":322,"threads_started":1}
Sep 29 16:42:35 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.65@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:42:35 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.65@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:42:35.792290 10078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:35.777122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:41429 (local address 127.0.197.65:33459)
0929 16:42:35.777702 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:35.777719 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:35.777908 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:35.778335 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:35.778340 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:35.778840 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:35.779105 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:35.779119 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:35.780208 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:35.780215 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:35.781156 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:35.781167 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:35.781867 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:35.781873 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:35.782046 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:35.786954 (+ 4908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:35.786979 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:35.791673 (+ 4694us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.197.126:41429: Server oryx/127.0.197.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":333,"threads_started":1}
W20260929 16:42:35.792362 10077 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:42:35.777276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:33459 (local address 127.0.197.126:41429)
0929 16:42:35.777541 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:35.777550 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:35.777830 (+ 280us) server_negotiation.cc:407] Connection header received
0929 16:42:35.778079 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:35.778084 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:35.778185 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:35.778303 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:35.779296 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:35.780060 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:35.781314 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:35.781712 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:35.781840 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:35.792121 (+ 10281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.197.65:33459: BlockingRecv error: recv got EOF from 127.0.197.65:33459 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Sep 29 16:42:36 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.66@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:42:36 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.66@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:42:36.138991 10079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:36.129787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:41429 (local address 127.0.197.66:47321)
0929 16:42:36.130330 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:36.130346 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:36.130505 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:36.130931 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:36.130935 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:36.131508 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:36.131680 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:36.131688 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:36.132713 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:36.132717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:36.133716 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:36.133728 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:36.134490 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:36.134494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:36.134645 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:36.135289 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:36.135307 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:36.138727 (+ 3420us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.197.126:41429: Server oryx/127.0.197.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":292,"threads_started":1}
W20260929 16:42:36.139284 10077 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:42:36.130102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:47321 (local address 127.0.197.126:41429)
0929 16:42:36.130312 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:36.130317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:36.130457 (+ 140us) server_negotiation.cc:407] Connection header received
0929 16:42:36.130667 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:36.130676 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:36.130738 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:36.130854 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:36.131842 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:36.132588 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:36.133840 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:36.134353 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:36.134448 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:36.138951 (+ 4503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.197.66:47321: BlockingRecv error: recv got EOF from 127.0.197.66:47321 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 29 16:42:36 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.67@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:42:36 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.67@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:42:36.673844 10080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:36.663985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:41429 (local address 127.0.197.67:47567)
0929 16:42:36.664771 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:36.664788 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:36.664894 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:36.665257 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:36.665260 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:36.665708 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:36.665858 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:36.665867 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:36.666909 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:36.666913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:36.667966 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:36.667980 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:36.668762 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:36.668767 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:36.668919 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:36.669547 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:36.669567 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:36.673587 (+ 4020us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.197.126:41429: Server oryx/127.0.197.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":537,"threads_started":1}
W20260929 16:42:36.674214 10081 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:42:36.664038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:47567 (local address 127.0.197.126:41429)
0929 16:42:36.664818 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:36.664823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:36.664839 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:42:36.665036 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:36.665040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:36.665120 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:36.665214 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:36.665995 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:36.666747 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:36.668139 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:36.668585 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:36.668696 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:36.673886 (+ 5190us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.197.67:47567: BlockingRecv error: recv got EOF from 127.0.197.67:47567 (error 108)
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":559,"threads_started":1}
Sep 29 16:42:36 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.65@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:42:36 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.65@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:42:36.804329 10082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:36.793458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:41429 (local address 127.0.197.65:50145)
0929 16:42:36.794080 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:36.794102 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:36.794303 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:36.794665 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:36.794670 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:36.795305 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:36.795520 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:36.795531 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:36.796744 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:36.796754 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:36.797817 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:36.797830 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:36.798658 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:36.798665 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:36.798863 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:36.799702 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:36.799721 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:36.803979 (+ 4258us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.197.126:41429: Server oryx/127.0.197.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":373,"threads_started":1}
W20260929 16:42:36.804435 10081 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:42:36.793752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:50145 (local address 127.0.197.126:41429)
0929 16:42:36.794058 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:36.794065 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:36.794215 (+ 150us) server_negotiation.cc:407] Connection header received
0929 16:42:36.794453 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:36.794459 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:36.794521 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:36.794635 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:36.795727 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:36.796577 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:36.798004 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:36.798470 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:36.798630 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:36.804201 (+ 5571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.197.65:50145: BlockingRecv error: recv got EOF from 127.0.197.65:50145 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
Sep 29 16:42:37 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.66@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:42:37 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.66@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:42:37.150087 10083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:37.140140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:41429 (local address 127.0.197.66:34475)
0929 16:42:37.140917 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:37.140935 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:37.141062 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:37.141377 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:37.141380 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:37.141824 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:37.142006 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:37.142014 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:37.143133 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:37.143137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:37.144205 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:37.144219 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:37.144931 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:37.144936 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:37.145090 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:37.145716 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:37.145737 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:37.149670 (+ 3933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.197.126:41429: Server oryx/127.0.197.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":535,"threads_started":1}
W20260929 16:42:37.150213 10081 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:42:37.140215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:34475 (local address 127.0.197.126:41429)
0929 16:42:37.140474 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:37.140482 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:37.141055 (+ 573us) server_negotiation.cc:407] Connection header received
0929 16:42:37.141166 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:37.141170 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:37.141242 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:37.141325 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:37.142140 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:37.142898 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:37.144341 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:37.144742 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:37.144844 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:37.149924 (+ 5080us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.197.66:34475: BlockingRecv error: recv got EOF from 127.0.197.66:34475 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Sep 29 16:42:37 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.67@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:42:37 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.67@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:42:37.684065 10084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:37.674889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:41429 (local address 127.0.197.67:39243)
0929 16:42:37.675587 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:37.675608 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:37.675732 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:37.676051 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:37.676054 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:37.676440 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:37.676606 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:37.676614 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:37.677625 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:37.677629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:37.678602 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:37.678624 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:37.679455 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:37.679460 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:37.679599 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:37.680233 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:37.680253 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:37.683823 (+ 3570us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.197.126:41429: Server oryx/127.0.197.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":475,"threads_started":1}
W20260929 16:42:37.684355 10085 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:42:37.674973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:39243 (local address 127.0.197.126:41429)
0929 16:42:37.675620 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:37.675632 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:37.675668 (+ 36us) server_negotiation.cc:407] Connection header received
0929 16:42:37.675838 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:37.675842 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:37.675909 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:37.675999 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:37.676731 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:37.677457 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:37.678764 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:37.679272 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:37.679378 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:37.684053 (+ 4675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.197.67:39243: BlockingRecv error: recv got EOF from 127.0.197.67:39243 (error 108)
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":352,"threads_started":1}
Sep 29 16:42:37 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.65@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:42:37 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.65@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:42:37.815862 10086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:37.805557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:41429 (local address 127.0.197.65:39313)
0929 16:42:37.806163 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:37.806186 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:37.806303 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:37.806617 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:37.806620 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:37.807127 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:37.807336 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:37.807345 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:37.808428 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:37.808433 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:37.809614 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:37.809626 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:37.810546 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:37.810551 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:37.810742 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:37.811605 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:37.811624 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:37.815594 (+ 3970us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.197.126:41429: Server oryx/127.0.197.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":303,"threads_started":1}
W20260929 16:42:37.816087 10085 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:42:37.805750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:39313 (local address 127.0.197.126:41429)
0929 16:42:37.805930 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:37.805934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:37.806307 (+ 373us) server_negotiation.cc:407] Connection header received
0929 16:42:37.806398 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:37.806403 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:37.806472 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:37.806579 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:37.807515 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:37.808266 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:37.809828 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:37.810336 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:37.810685 (+ 349us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:37.815826 (+ 5141us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.197.65:39313: BlockingRecv error: recv got EOF from 127.0.197.65:39313 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 29 16:42:38 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.66@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:42:38 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.66@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:42:38.161543 10087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:38.151473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:41429 (local address 127.0.197.66:40797)
0929 16:42:38.152105 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:38.152123 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:38.152247 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:38.152718 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:38.152722 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:38.153121 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:38.153290 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:38.153299 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:38.154502 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:38.154510 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:38.155946 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:38.155965 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:38.156680 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:38.156685 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:38.156796 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:38.157404 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:38.157422 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:38.161270 (+ 3848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.197.126:41429: Server oryx/127.0.197.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":315,"threads_started":1}
W20260929 16:42:38.161839 10085 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:42:38.151767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:40797 (local address 127.0.197.126:41429)
0929 16:42:38.152001 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:38.152006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:38.152209 (+ 203us) server_negotiation.cc:407] Connection header received
0929 16:42:38.152425 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:38.152438 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:38.152514 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:38.152636 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:38.153468 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:38.154298 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:38.156110 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:38.156539 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:38.156640 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:38.161531 (+ 4891us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.197.66:40797: BlockingRecv error: recv got EOF from 127.0.197.66:40797 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
Sep 29 16:42:38 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.67@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:42:38 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.67@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:42:38.698266 10088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:38.685180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:41429 (local address 127.0.197.67:59469)
0929 16:42:38.685823 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:38.685841 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:38.685962 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:38.688072 (+ 2110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:38.688080 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:38.688500 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:38.688767 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:38.688778 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:38.690511 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:38.690516 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:38.691919 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:38.691934 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:38.693040 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:38.693045 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:38.693169 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:38.693877 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:38.693896 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:38.698004 (+ 4108us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.197.126:41429: Server oryx/127.0.197.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":329,"threads_started":1}
W20260929 16:42:38.698750 10089 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:42:38.685665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:59469 (local address 127.0.197.126:41429)
0929 16:42:38.687581 (+ 1916us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:38.687587 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:38.687610 (+ 23us) server_negotiation.cc:407] Connection header received
0929 16:42:38.687675 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:38.687679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:38.687769 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:38.688025 (+ 256us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:38.688947 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:38.690322 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:38.692105 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:38.692822 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:38.692957 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:38.698298 (+ 5341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.197.67:59469: BlockingRecv error: recv got EOF from 127.0.197.67:59469 (error 108)
Metrics: {"server-negotiator.queue_time_us":1710,"thread_start_us":1638,"threads_started":1}
Sep 29 16:42:38 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.65@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:42:38 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.65@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:42:38.826785 10090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:38.816843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:41429 (local address 127.0.197.65:37685)
0929 16:42:38.817482 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:38.817499 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:38.817617 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:38.817930 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:38.817933 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:38.818326 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:38.818503 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:38.818512 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:38.819831 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:38.819836 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:38.820902 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:38.820917 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:38.821654 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:38.821658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:38.821778 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:38.822442 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:38.822461 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:38.826284 (+ 3823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.197.126:41429: Server oryx/127.0.197.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":386,"threads_started":1}
W20260929 16:42:38.826830 10089 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:42:38.817006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:37685 (local address 127.0.197.126:41429)
0929 16:42:38.817182 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:38.817187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:38.817579 (+ 392us) server_negotiation.cc:407] Connection header received
0929 16:42:38.817729 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:38.817733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:38.817792 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:38.817879 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:38.818640 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:38.819609 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:38.821064 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:38.821486 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:38.821591 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:38.826569 (+ 4978us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.197.65:37685: BlockingRecv error: recv got EOF from 127.0.197.65:37685 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 29 16:42:39 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.66@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:42:39 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.66@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
W20260929 16:42:39.172319 10089 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:42:39.162745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:44443 (local address 127.0.197.126:41429)
0929 16:42:39.162931 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:39.162935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:39.163311 (+ 376us) server_negotiation.cc:407] Connection header received
0929 16:42:39.163481 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:39.163485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:39.163546 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:39.163640 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:39.164437 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:39.165481 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:39.166788 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:39.167239 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:39.167345 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:39.172018 (+ 4673us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.197.66:44443: BlockingRecv error: recv got EOF from 127.0.197.66:44443 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 16:42:39.172340 10091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:39.162639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:41429 (local address 127.0.197.66:44443)
0929 16:42:39.163223 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:39.163239 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:39.163356 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:39.163677 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:39.163681 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:39.164075 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:39.164264 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:39.164273 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:39.165661 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:39.165666 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:39.166603 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:39.166617 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:39.167418 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:39.167423 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:39.167543 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:39.168178 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:39.168195 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:39.171693 (+ 3498us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.197.126:41429: Server oryx/127.0.197.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":279,"threads_started":1}
Sep 29 16:42:39 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.67@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:42:39 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.67@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:42:39.709478 10092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:39.699552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:41429 (local address 127.0.197.67:45765)
0929 16:42:39.700082 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:39.700100 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:39.700222 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:39.700745 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:39.700748 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:39.701149 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:39.701314 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:39.701322 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:39.702395 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:39.702399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:39.703497 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:39.703537 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:39.704419 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:39.704425 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:39.704561 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:39.705288 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:39.705323 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:39.709157 (+ 3834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.197.126:41429: Server oryx/127.0.197.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":263,"threads_started":1}
W20260929 16:42:39.709837 10093 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:42:39.699645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:45765 (local address 127.0.197.126:41429)
0929 16:42:39.700408 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:39.700414 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:39.700447 (+ 33us) server_negotiation.cc:407] Connection header received
0929 16:42:39.700531 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:39.700536 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:39.700601 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:39.700694 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:39.701479 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:39.702236 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:39.703707 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:39.704187 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:39.704301 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:39.709473 (+ 5172us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.197.67:45765: BlockingRecv error: recv got EOF from 127.0.197.67:45765 (error 108)
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":530,"threads_started":1}
Sep 29 16:42:39 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.65@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:42:39 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.65@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:42:39.839915 10094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:39.827994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:41429 (local address 127.0.197.65:45355)
0929 16:42:39.828635 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:39.828653 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:39.829009 (+ 356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:39.829440 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:39.829443 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:39.829855 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:39.830006 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:39.830014 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:39.831180 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:39.831184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:39.832322 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:39.832337 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:39.833391 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:39.833396 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:39.833531 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:39.834218 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:39.834237 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:39.839641 (+ 5404us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.197.126:41429: Server oryx/127.0.197.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":359,"threads_started":1}
W20260929 16:42:39.840070 10093 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:42:39.828132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:45355 (local address 127.0.197.126:41429)
0929 16:42:39.828314 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:39.828319 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:39.828975 (+ 656us) server_negotiation.cc:407] Connection header received
0929 16:42:39.829127 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:39.829131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:39.829207 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:39.829313 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:39.830169 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:39.830984 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:39.832517 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:39.833222 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:39.833342 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:39.839864 (+ 6522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.197.65:45355: BlockingRecv error: recv got EOF from 127.0.197.65:45355 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 29 16:42:40 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.66@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:42:40 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.66@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:42:40.184460 10095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:40.173795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:41429 (local address 127.0.197.66:48363)
0929 16:42:40.174394 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:40.174410 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:40.174592 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:40.175047 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:40.175055 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:40.175495 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:40.175638 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:40.175675 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:40.176778 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:40.176786 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:40.178254 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:40.178275 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:40.179440 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:40.179452 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:40.179599 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:40.180262 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:40.180281 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:40.184210 (+ 3929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.197.126:41429: Server oryx/127.0.197.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":342,"threads_started":1}
W20260929 16:42:40.184823 10093 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:42:40.173927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:48363 (local address 127.0.197.126:41429)
0929 16:42:40.174211 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:40.174218 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:40.174510 (+ 292us) server_negotiation.cc:407] Connection header received
0929 16:42:40.174762 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:40.174766 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:40.174879 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:40.174982 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:40.175817 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:40.176580 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:40.178512 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:40.178992 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:40.179175 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:40.184434 (+ 5259us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.197.66:48363: BlockingRecv error: recv got EOF from 127.0.197.66:48363 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Sep 29 16:42:40 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.67@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:42:40 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.67@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:42:40.720999 10096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:40.710718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:41429 (local address 127.0.197.67:48693)
0929 16:42:40.711395 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:40.711417 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:40.711578 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:40.711978 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:40.711985 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:40.712545 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:40.712732 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:40.712742 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:40.713771 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:40.713776 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:40.714968 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:40.714982 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:40.716119 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:40.716126 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:40.716320 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:40.716958 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:40.716992 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:40.720751 (+ 3759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.197.126:41429: Server oryx/127.0.197.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":311,"threads_started":1}
W20260929 16:42:40.721259 10097 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:42:40.710753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:48693 (local address 127.0.197.126:41429)
0929 16:42:40.711580 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:40.711587 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:40.711606 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:42:40.711694 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:40.711699 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:40.711796 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:40.711935 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:40.712878 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:40.713614 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:40.715219 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:40.715907 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:40.716099 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:40.721024 (+ 4925us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.197.67:48693: BlockingRecv error: recv got EOF from 127.0.197.67:48693 (error 108)
Metrics: {"server-negotiator.queue_time_us":712,"thread_start_us":582,"threads_started":1}
Sep 29 16:42:40 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.65@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:42:40 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.65@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:42:40.857611 10098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:40.841078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:41429 (local address 127.0.197.65:44213)
0929 16:42:40.841681 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:40.841697 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:40.841814 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:40.842151 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:40.842155 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:40.842537 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:40.842697 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:40.842706 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:40.843963 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:40.843968 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:40.844971 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:40.844984 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:40.845793 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:40.845798 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:40.845919 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:40.853106 (+ 7187us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:40.853164 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:40.857327 (+ 4163us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.197.126:41429: Server oryx/127.0.197.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":403,"threads_started":1}
W20260929 16:42:40.857820 10097 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:42:40.841171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:44213 (local address 127.0.197.126:41429)
0929 16:42:40.841364 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:40.841368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:40.841798 (+ 430us) server_negotiation.cc:407] Connection header received
0929 16:42:40.841918 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:40.841922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:40.841982 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:40.842067 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:40.842905 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:40.843792 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:40.845133 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:40.845561 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:40.845767 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:40.857595 (+ 11828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.197.65:44213: BlockingRecv error: recv got EOF from 127.0.197.65:44213 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 29 16:42:41 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.66@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:42:41 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.66@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:42:41.197924 10099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:41.185561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:41429 (local address 127.0.197.66:60919)
0929 16:42:41.186148 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:41.186172 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:41.186322 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:41.186695 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:41.186701 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:41.187331 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:41.187600 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:41.187615 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:41.188697 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:41.188706 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:41.190660 (+ 1954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:41.190680 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:41.191520 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:41.191529 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:41.191755 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:41.192472 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:41.192502 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:41.197678 (+ 5176us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.197.126:41429: Server oryx/127.0.197.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":288,"threads_started":1}
W20260929 16:42:41.198331 10097 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:42:41.185637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:60919 (local address 127.0.197.126:41429)
0929 16:42:41.185878 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:41.185885 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:41.186261 (+ 376us) server_negotiation.cc:407] Connection header received
0929 16:42:41.186462 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:41.186465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:41.186544 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:41.186627 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:41.187760 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:41.188542 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:41.190847 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:41.191350 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:41.191449 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:41.197992 (+ 6543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.197.66:60919: BlockingRecv error: recv got EOF from 127.0.197.66:60919 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 29 16:42:41 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.67@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:42:41 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.67@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:42:41.731953 10100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:41.722043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:41429 (local address 127.0.197.67:43017)
0929 16:42:41.722578 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:41.722595 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:41.722707 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:41.723185 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:41.723188 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:41.723534 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:41.723709 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:41.723717 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:41.724841 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:41.724846 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:41.726246 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:41.726260 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:41.727191 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:41.727195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:41.727311 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:41.727976 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:41.727995 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:41.731573 (+ 3578us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.197.126:41429: Server oryx/127.0.197.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":289,"threads_started":1}
W20260929 16:42:41.732158 10101 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:42:41.722287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:43017 (local address 127.0.197.126:41429)
0929 16:42:41.722781 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:41.722786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:41.722813 (+ 27us) server_negotiation.cc:407] Connection header received
0929 16:42:41.722866 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:41.722870 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:41.722939 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:41.723152 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:41.723850 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:41.724693 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:41.726406 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:41.726975 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:41.727162 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:41.731806 (+ 4644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.197.67:43017: BlockingRecv error: recv got EOF from 127.0.197.67:43017 (error 108)
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":285,"threads_started":1}
Sep 29 16:42:41 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.65@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:42:41 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.65@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:42:41.870303 10102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:41.858774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:41429 (local address 127.0.197.65:60755)
0929 16:42:41.859485 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:41.859502 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:41.859632 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:41.860278 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:41.860282 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:41.860724 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:41.860912 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:41.860920 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:41.862206 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:41.862210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:41.863164 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:41.863177 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:41.864066 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:41.864072 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:41.864211 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:41.864962 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:41.864980 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:41.870069 (+ 5089us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.197.126:41429: Server oryx/127.0.197.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":476,"threads_started":1}
W20260929 16:42:41.870524 10101 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:42:41.858995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:60755 (local address 127.0.197.126:41429)
0929 16:42:41.859218 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:41.859225 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:41.859599 (+ 374us) server_negotiation.cc:407] Connection header received
0929 16:42:41.859971 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:41.859977 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:41.860074 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:41.860211 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:41.861128 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:41.862052 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:41.863372 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:41.863826 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:41.863944 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:41.870303 (+ 6359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.197.65:60755: BlockingRecv error: recv got EOF from 127.0.197.65:60755 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 29 16:42:42 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.66@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:42:42 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.66@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:42:42.209771 10103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:42.199567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:41429 (local address 127.0.197.66:35899)
0929 16:42:42.200104 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:42.200121 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:42.200363 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:42.200799 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:42.200803 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:42.201394 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:42.201655 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:42.201669 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:42.202985 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:42.202990 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:42.204032 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:42.204046 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:42.204762 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:42.204767 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:42.204912 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:42.205527 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:42.205546 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:42.209302 (+ 3756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.197.126:41429: Server oryx/127.0.197.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":253,"threads_started":1}
W20260929 16:42:42.209833 10101 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:42:42.199768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:35899 (local address 127.0.197.126:41429)
0929 16:42:42.199966 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:42.199973 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:42.200216 (+ 243us) server_negotiation.cc:407] Connection header received
0929 16:42:42.200517 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:42.200522 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:42.200602 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:42.200706 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:42.201848 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:42.202813 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:42.204195 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:42.204585 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:42.204685 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:42.209555 (+ 4870us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.197.66:35899: BlockingRecv error: recv got EOF from 127.0.197.66:35899 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 29 16:42:42 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.67@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:42:42 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.67@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:42:42.742544 10104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:42.732940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:41429 (local address 127.0.197.67:48807)
0929 16:42:42.733510 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:42.733533 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:42.733667 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:42.734034 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:42.734039 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:42.734487 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:42.734677 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:42.734687 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:42.735821 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:42.735828 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:42.736741 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:42.736754 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:42.737536 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:42.737542 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:42.737726 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:42.738358 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:42.738376 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:42.742201 (+ 3825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.197.126:41429: Server oryx/127.0.197.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":299,"threads_started":1}
W20260929 16:42:42.742722 10105 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:42:42.733051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:48807 (local address 127.0.197.126:41429)
0929 16:42:42.733679 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:42.733685 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:42.733703 (+ 18us) server_negotiation.cc:407] Connection header received
0929 16:42:42.733795 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:42.733800 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:42.733886 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:42.734007 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:42.734833 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:42.735684 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:42.736898 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:42.737339 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:42.737496 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:42.742444 (+ 4948us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.197.67:48807: BlockingRecv error: recv got EOF from 127.0.197.67:48807 (error 108)
Metrics: {"server-negotiator.queue_time_us":527,"spinlock_wait_cycles":7936,"thread_start_us":509,"threads_started":1}
Sep 29 16:42:42 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.65@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:42:42 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.65@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:42:42.880750 10106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:42.871362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:41429 (local address 127.0.197.65:44519)
0929 16:42:42.871928 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:42.871944 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:42.872093 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:42.872483 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:42.872488 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:42.872984 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:42.873153 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:42.873163 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:42.874205 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:42.874208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:42.875166 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:42.875177 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:42.875913 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:42.875917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:42.876049 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:42.876763 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:42.876782 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:42.880488 (+ 3706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.197.126:41429: Server oryx/127.0.197.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":339,"threads_started":1}
W20260929 16:42:42.880966 10105 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:42:42.871402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:44519 (local address 127.0.197.126:41429)
0929 16:42:42.871636 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:42.871642 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:42.872051 (+ 409us) server_negotiation.cc:407] Connection header received
0929 16:42:42.872217 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:42.872221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:42.872314 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:42.872423 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:42.873304 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:42.874064 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:42.875343 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:42.875774 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:42.875878 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:42.880734 (+ 4856us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.197.65:44519: BlockingRecv error: recv got EOF from 127.0.197.65:44519 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 29 16:42:43 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.66@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:42:43 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.66@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:42:43.221880 10107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:43.211316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:41429 (local address 127.0.197.66:46275)
0929 16:42:43.212440 (+ 1124us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:43.212469 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:43.212679 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:43.213110 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:43.213115 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:43.213658 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:43.213827 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:43.213836 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:43.215087 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:43.215094 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:43.216026 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:43.216040 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:43.216783 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:43.216789 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:43.216985 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:43.217646 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:43.217666 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:43.221468 (+ 3802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.197.126:41429: Server oryx/127.0.197.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":960,"thread_start_us":800,"threads_started":1}
W20260929 16:42:43.221964 10105 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:42:43.211329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:46275 (local address 127.0.197.126:41429)
0929 16:42:43.211631 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:43.211643 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:43.212608 (+ 965us) server_negotiation.cc:407] Connection header received
0929 16:42:43.212826 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:43.212832 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:43.212939 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:43.213084 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:43.213983 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:43.214905 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:43.216182 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:43.216623 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:43.216763 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:43.221717 (+ 4954us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.197.66:46275: BlockingRecv error: recv got EOF from 127.0.197.66:46275 (error 108)
Metrics: {"server-negotiator.queue_time_us":124}
Sep 29 16:42:43 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.67@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:42:43 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.67@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:42:43.753746 10108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:43.743774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:41429 (local address 127.0.197.67:42729)
0929 16:42:43.744778 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:43.744796 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:43.744983 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:43.745415 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:43.745420 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:43.745834 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:43.746001 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:43.746010 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:43.747133 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:43.747137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:43.748074 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:43.748086 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:43.748842 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:43.748846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:43.748969 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:43.749577 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:43.749595 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:43.753371 (+ 3776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.197.126:41429: Server oryx/127.0.197.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":496,"threads_started":1}
W20260929 16:42:43.753914 10109 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:42:43.743856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:42729 (local address 127.0.197.126:41429)
0929 16:42:43.744791 (+ 935us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:43.744799 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:43.744950 (+ 151us) server_negotiation.cc:407] Connection header received
0929 16:42:43.745134 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:43.745139 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:43.745253 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:43.745367 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:43.746146 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:43.746927 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:43.748230 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:43.748679 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:43.748786 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:43.753631 (+ 4845us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.197.67:42729: BlockingRecv error: recv got EOF from 127.0.197.67:42729 (error 108)
Metrics: {"server-negotiator.queue_time_us":817,"thread_start_us":716,"threads_started":1}
Sep 29 16:42:43 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.65@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:42:43 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.65@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:42:43.892210 10110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:43.881910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:41429 (local address 127.0.197.65:59273)
0929 16:42:43.882582 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:43.882615 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:43.882762 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:43.883154 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:43.883159 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:43.883690 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:43.883838 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:43.883847 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:43.885178 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:43.885183 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:43.886262 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:43.886275 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:43.886957 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:43.886963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:43.887177 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:43.888036 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:43.888058 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:43.891839 (+ 3781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.197.126:41429: Server oryx/127.0.197.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":369,"threads_started":1}
W20260929 16:42:43.892331 10109 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:42:43.882094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:59273 (local address 127.0.197.126:41429)
0929 16:42:43.882393 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:43.882399 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:43.882780 (+ 381us) server_negotiation.cc:407] Connection header received
0929 16:42:43.882888 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:43.882894 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:43.882987 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:43.883107 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:43.883985 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:43.885017 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:43.886405 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:43.886812 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:43.886916 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:43.892095 (+ 5179us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.197.65:59273: BlockingRecv error: recv got EOF from 127.0.197.65:59273 (error 108)
Metrics: {"server-negotiator.queue_time_us":130}
Sep 29 16:42:44 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.66@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:42:44 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.66@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:42:44.233326 10111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:44.223132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:41429 (local address 127.0.197.66:49965)
0929 16:42:44.223693 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:44.223715 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:44.223850 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:44.224273 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:44.224277 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:44.224872 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:44.225034 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:44.225044 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:44.226243 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:44.226248 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:44.227293 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:44.227307 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:44.228073 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:44.228078 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:44.228214 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:44.228848 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:44.228866 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:44.233067 (+ 4201us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.197.126:41429: Server oryx/127.0.197.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":312,"threads_started":1}
W20260929 16:42:44.233661 10109 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:42:44.223480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:49965 (local address 127.0.197.126:41429)
0929 16:42:44.223719 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:44.223724 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:44.223789 (+ 65us) server_negotiation.cc:407] Connection header received
0929 16:42:44.224001 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:44.224007 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:44.224097 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:44.224198 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:44.225172 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:44.226087 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:44.227472 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:44.227907 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:44.228015 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:44.233326 (+ 5311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.197.66:49965: BlockingRecv error: recv got EOF from 127.0.197.66:49965 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 29 16:42:44 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.67@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:42:44 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.67@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:42:44.765781 10112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:44.755254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:41429 (local address 127.0.197.67:33229)
0929 16:42:44.755948 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:44.755969 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:44.756122 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:44.756804 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:44.756811 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:44.757392 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:44.757605 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:44.757618 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:44.759111 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:44.759115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:44.760073 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:44.760087 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:44.760840 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:44.760844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:44.760974 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:44.761801 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:44.761820 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:44.765525 (+ 3705us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.197.126:41429: Server oryx/127.0.197.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":310,"threads_started":1}
W20260929 16:42:44.766144 10113 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:42:44.755373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:33229 (local address 127.0.197.126:41429)
0929 16:42:44.756274 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:44.756279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:44.756435 (+ 156us) server_negotiation.cc:407] Connection header received
0929 16:42:44.756495 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:44.756500 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:44.756611 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:44.756713 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:44.757785 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:44.758848 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:44.760254 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:44.760697 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:44.760806 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:44.765831 (+ 5025us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.197.67:33229: BlockingRecv error: recv got EOF from 127.0.197.67:33229 (error 108)
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":475,"threads_started":1}
Sep 29 16:42:44 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.65@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:42:44 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.65@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:42:44.903688 10114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:44.893505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:41429 (local address 127.0.197.65:47275)
0929 16:42:44.894042 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:44.894059 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:44.894183 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:44.894513 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:44.894517 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:44.894883 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:44.895153 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:44.895164 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:44.896387 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:44.896391 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:44.897678 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:44.897694 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:44.898504 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:44.898509 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:44.898642 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:44.899280 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:44.899298 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:44.903400 (+ 4102us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.197.126:41429: Server oryx/127.0.197.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":299,"threads_started":1}
W20260929 16:42:44.903927 10113 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:42:44.893662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:47275 (local address 127.0.197.126:41429)
0929 16:42:44.893933 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:44.893937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:44.894174 (+ 237us) server_negotiation.cc:407] Connection header received
0929 16:42:44.894286 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:44.894290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:44.894356 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:44.894492 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:44.895318 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:44.896240 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:44.897859 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:44.898369 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:44.898471 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:44.903692 (+ 5221us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.197.65:47275: BlockingRecv error: recv got EOF from 127.0.197.65:47275 (error 108)
Metrics: {"server-negotiator.queue_time_us":160}
Sep 29 16:42:45 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.66@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 16:42:45 dist-test-slave-l31f krb5kdc[9053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.197.66@KRBTEST.COM for oryx/127.0.197.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 16:42:45.244485 10115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:45.234783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:41429 (local address 127.0.197.66:56997)
0929 16:42:45.235358 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:45.235374 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:45.235504 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:45.235816 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:45.235819 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:45.236192 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:45.236365 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:45.236374 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:45.237409 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:45.237414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:45.238395 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:45.238407 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:45.239425 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:45.239430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:45.239691 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:45.240483 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:45.240502 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:45.244202 (+ 3700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.197.126:41429: Server oryx/127.0.197.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":324,"threads_started":1}
W20260929 16:42:45.244726 10113 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:42:45.234902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:56997 (local address 127.0.197.126:41429)
0929 16:42:45.235168 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:45.235173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:45.235465 (+ 292us) server_negotiation.cc:407] Connection header received
0929 16:42:45.235614 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:45.235618 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:45.235679 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:45.235775 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:45.236512 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:45.237249 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:45.238558 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:45.239273 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:45.239407 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:45.244445 (+ 5038us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.197.66:56997: BlockingRecv error: recv got EOF from 127.0.197.66:56997 (error 108)
Metrics: {"server-negotiator.queue_time_us":125}
I20260929 16:42:45.325450 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 9642
I20260929 16:42:45.339828 10117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:45.326850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47402 (local address 127.0.197.65:34557)
0929 16:42:45.327593 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:45.327601 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:45.327632 (+ 31us) server_negotiation.cc:407] Connection header received
0929 16:42:45.327698 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:45.327705 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:45.327946 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:45.328110 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:45.329728 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:45.330867 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:45.332260 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:45.333017 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:45.333133 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:45.335930 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:45.335941 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:45.335978 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:45.336033 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:45.339217 (+ 3184us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.197.65@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 16:42:45.339575 (+ 358us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47402: Request ticket server kudu/127.0.197.65@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":420,"threads_started":1}
W20260929 16:42:45.339986 10117 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47402: Request ticket server kudu/127.0.197.65@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 16:42:45.340147 789 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 9642: Not authorized: Client connection negotiation failed: client connection to 127.0.197.65:34557: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.197.65@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 16:42:45.340399 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 9642
I20260929 16:42:45.358145 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 9793
I20260929 16:42:45.371733 10118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:45.359492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53216 (local address 127.0.197.66:33143)
0929 16:42:45.360261 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:45.360270 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:45.360308 (+ 38us) server_negotiation.cc:407] Connection header received
0929 16:42:45.360393 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:45.360405 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:45.360738 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:45.360984 (+ 246us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:45.362281 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:45.363499 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:45.364851 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:45.365501 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:45.365620 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:45.368340 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:45.368350 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:45.368365 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:45.368415 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:45.371123 (+ 2708us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.197.66@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 16:42:45.371467 (+ 344us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53216: Request ticket server kudu/127.0.197.66@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":410,"threads_started":1}
W20260929 16:42:45.371826 10118 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53216: Request ticket server kudu/127.0.197.66@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 16:42:45.372156 789 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 9793: Not authorized: Client connection negotiation failed: client connection to 127.0.197.66:33143: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.197.66@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 16:42:45.372435 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 9793
I20260929 16:42:45.389796 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 9926
I20260929 16:42:45.405046 10119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:45.391638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43572 (local address 127.0.197.67:35167)
0929 16:42:45.392210 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:45.392218 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:45.392244 (+ 26us) server_negotiation.cc:407] Connection header received
0929 16:42:45.392306 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:45.392312 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:45.392548 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:45.392771 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:45.394170 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:45.395521 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:45.396929 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:45.397560 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:45.397746 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:45.401020 (+ 3274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:45.401031 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:45.401047 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:45.401096 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:45.404255 (+ 3159us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.197.67@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 16:42:45.404678 (+ 423us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43572: Request ticket server kudu/127.0.197.67@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":287,"threads_started":1}
W20260929 16:42:45.405212 10119 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43572: Request ticket server kudu/127.0.197.67@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 16:42:45.405234 789 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 9926: Not authorized: Client connection negotiation failed: client connection to 127.0.197.67:35167: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.197.67@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 16:42:45.405400 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 9926
I20260929 16:42:45.423163 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 9573
I20260929 16:42:45.424593 9635 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:42:45.550652 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 9573
2026-09-29T16:42:45Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21716 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-29T16:42:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:42:45Z Disabled control of system clock
I20260929 16:42:45.633283 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:45001
--webserver_interface=127.0.197.126
--webserver_port=0
--builtin_ntp_servers=127.0.197.84:46549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.126:45001
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 16:42:46.009295 10124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:42:46.009637 10124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:42:46.009718 10124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:42:46.017328 10124 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:42:46.017459 10124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:42:46.017498 10124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:42:46.017551 10124 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:42:46.017587 10124 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:42:46.027127 10124 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:46549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.197.126:45001
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:45001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.197.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:42:46.028456 10124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:42:46.030310 10124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:42:46.047915 10132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:46.047976 10129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:46.047996 10130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:42:46.048257 10124 server_base.cc:1034] running on GCE node
I20260929 16:42:46.049275 10124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:42:46.050848 10124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:42:46.052145 10124 hybrid_clock.cc:648] HybridClock initialized: now 1790700166052080 us; error 83 us; skew 500 ppm
I20260929 16:42:46.052558 10124 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:42:46.062810 10124 webserver.cc:460] Webserver started at http://127.0.197.126:41623/ using document root <none> and password file <none>
I20260929 16:42:46.063745 10124 fs_manager.cc:359] Metadata directory not provided
I20260929 16:42:46.063905 10124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:42:46.064265 10124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:42:46.067451 10124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/master-0/data/instance:
uuid: "4ce1b2d9f82e452a8c09bc225083265a"
format_stamp: "Formatted at 2026-09-29 16:42:46 on dist-test-slave-l31f"
server_key: "a16e368ce02cc2bae8fc413464b23c7c"
server_key_iv: "8b48b7264b216ab40b703af56e79f876"
server_key_version: "encryptionkey@0"
I20260929 16:42:46.068392 10124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/master-0/wal/instance:
uuid: "4ce1b2d9f82e452a8c09bc225083265a"
format_stamp: "Formatted at 2026-09-29 16:42:46 on dist-test-slave-l31f"
server_key: "a16e368ce02cc2bae8fc413464b23c7c"
server_key_iv: "8b48b7264b216ab40b703af56e79f876"
server_key_version: "encryptionkey@0"
I20260929 16:42:46.074296 10124 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 16:42:46.078769 10138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:46.080519 10124 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:42:46.080773 10124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "4ce1b2d9f82e452a8c09bc225083265a"
format_stamp: "Formatted at 2026-09-29 16:42:46 on dist-test-slave-l31f"
server_key: "a16e368ce02cc2bae8fc413464b23c7c"
server_key_iv: "8b48b7264b216ab40b703af56e79f876"
server_key_version: "encryptionkey@0"
I20260929 16:42:46.081032 10124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:42:46.111141 10124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:42:46.112121 10124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:42:46.112460 10124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:42:46.134728 10124 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.126:45001
I20260929 16:42:46.134752 10189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.126:45001 every 8 connection(s)
I20260929 16:42:46.136327 10124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:42:46.139570 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 10124
I20260929 16:42:46.139812 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/master-0/wal/instance
I20260929 16:42:46.140551 789 external_mini_cluster.cc:1428] Setting key 8b441ca6ca06e890c2d66b1e4e981656
I20260929 16:42:46.140595 10190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:42:46.157984 10190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ce1b2d9f82e452a8c09bc225083265a: Bootstrap starting.
I20260929 16:42:46.158676 10193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:46.142827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35688 (local address 127.0.197.126:45001)
0929 16:42:46.144438 (+ 1611us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:46.144464 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:46.144532 (+ 68us) server_negotiation.cc:407] Connection header received
0929 16:42:46.147539 (+ 3007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:46.147577 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:46.148218 (+ 641us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:46.148698 (+ 480us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:46.149360 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:46.150739 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:46.152405 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:46.153722 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:46.153984 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:46.155417 (+ 1433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:46.155429 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:46.155454 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:42:46.156253 (+ 799us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:46.156379 (+ 126us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:42:46.156414 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:46.156560 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:46.156880 (+ 320us) server_negotiation.cc:299] Negotiation successful
0929 16:42:46.157282 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":903,"thread_start_us":569,"threads_started":1}
I20260929 16:42:46.162236 10190 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ce1b2d9f82e452a8c09bc225083265a: Neither blocks nor log segments found. Creating new log.
I20260929 16:42:46.163640 10190 log.cc:824] T 00000000000000000000000000000000 P 4ce1b2d9f82e452a8c09bc225083265a: Log is configured to *not* fsync() on all Append() calls
I20260929 16:42:46.168021 10190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ce1b2d9f82e452a8c09bc225083265a: No bootstrap required, opened a new log
I20260929 16:42:46.177466 10190 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ce1b2d9f82e452a8c09bc225083265a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ce1b2d9f82e452a8c09bc225083265a" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 45001 } }
I20260929 16:42:46.177906 10190 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ce1b2d9f82e452a8c09bc225083265a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:42:46.177978 10190 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ce1b2d9f82e452a8c09bc225083265a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ce1b2d9f82e452a8c09bc225083265a, State: Initialized, Role: FOLLOWER
I20260929 16:42:46.178567 10190 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ce1b2d9f82e452a8c09bc225083265a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ce1b2d9f82e452a8c09bc225083265a" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 45001 } }
I20260929 16:42:46.178804 10190 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ce1b2d9f82e452a8c09bc225083265a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:42:46.178902 10190 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ce1b2d9f82e452a8c09bc225083265a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:42:46.179138 10190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ce1b2d9f82e452a8c09bc225083265a [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:42:46.182168 10190 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ce1b2d9f82e452a8c09bc225083265a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ce1b2d9f82e452a8c09bc225083265a" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 45001 } }
I20260929 16:42:46.182919 10190 leader_election.cc:302] T 00000000000000000000000000000000 P 4ce1b2d9f82e452a8c09bc225083265a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ce1b2d9f82e452a8c09bc225083265a; no voters:
I20260929 16:42:46.183670 10190 leader_election.cc:288] T 00000000000000000000000000000000 P 4ce1b2d9f82e452a8c09bc225083265a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:42:46.183880 10195 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ce1b2d9f82e452a8c09bc225083265a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:42:46.184572 10195 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ce1b2d9f82e452a8c09bc225083265a [term 1 LEADER]: Becoming Leader. State: Replica: 4ce1b2d9f82e452a8c09bc225083265a, State: Running, Role: LEADER
I20260929 16:42:46.185235 10195 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ce1b2d9f82e452a8c09bc225083265a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ce1b2d9f82e452a8c09bc225083265a" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 45001 } }
I20260929 16:42:46.186058 10190 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ce1b2d9f82e452a8c09bc225083265a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:42:46.188375 10196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ce1b2d9f82e452a8c09bc225083265a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ce1b2d9f82e452a8c09bc225083265a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ce1b2d9f82e452a8c09bc225083265a" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 45001 } } }
I20260929 16:42:46.188776 10196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ce1b2d9f82e452a8c09bc225083265a [sys.catalog]: This master's current role is: LEADER
I20260929 16:42:46.189684 10200 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:42:46.189544 10197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ce1b2d9f82e452a8c09bc225083265a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ce1b2d9f82e452a8c09bc225083265a. Latest consensus state: current_term: 1 leader_uuid: "4ce1b2d9f82e452a8c09bc225083265a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ce1b2d9f82e452a8c09bc225083265a" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 45001 } } }
I20260929 16:42:46.189908 10197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ce1b2d9f82e452a8c09bc225083265a [sys.catalog]: This master's current role is: LEADER
I20260929 16:42:46.195691 10200 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:42:46.207647 10200 catalog_manager.cc:1347] Generated new cluster ID: 37361d2f2fa44f7d975fb389f3482c18
I20260929 16:42:46.207798 10200 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:42:46.224961 10200 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:42:46.226346 10200 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:42:46.238601 10200 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ce1b2d9f82e452a8c09bc225083265a: Generated new TSK 0
I20260929 16:42:46.239585 10200 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:42:46.245509 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.65:0
--local_ip_for_outbound_sockets=127.0.197.65
--webserver_interface=127.0.197.65
--webserver_port=0
--tserver_master_addrs=127.0.197.126:45001
--builtin_ntp_servers=127.0.197.84:46549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 16:42:46.614866 10214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:42:46.615275 10214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:42:46.615438 10214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:42:46.623521 10214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:42:46.623669 10214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:42:46.623843 10214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.65
I20260929 16:42:46.633699 10214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:46549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.197.65
--webserver_port=0
--tserver_master_addrs=127.0.197.126:45001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.65
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:42:46.635186 10214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:42:46.637058 10214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:42:46.650890 10219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:46.650962 10222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:46.650938 10220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:42:46.652282 10214 server_base.cc:1034] running on GCE node
I20260929 16:42:46.652930 10214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:42:46.653985 10214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:42:46.655193 10214 hybrid_clock.cc:648] HybridClock initialized: now 1790700166655133 us; error 50 us; skew 500 ppm
I20260929 16:42:46.655517 10214 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:42:46.658412 10214 webserver.cc:460] Webserver started at http://127.0.197.65:37181/ using document root <none> and password file <none>
I20260929 16:42:46.659260 10214 fs_manager.cc:359] Metadata directory not provided
I20260929 16:42:46.659415 10214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:42:46.659832 10214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:42:46.663278 10214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/ts-0/data/instance:
uuid: "023022f19f764ad792f6bb884b224151"
format_stamp: "Formatted at 2026-09-29 16:42:46 on dist-test-slave-l31f"
server_key: "ab459213a0c872b5b702cfde7d3f6abc"
server_key_iv: "977d53b3d20a7fd3b6a9bd8824b227ce"
server_key_version: "encryptionkey@0"
I20260929 16:42:46.664300 10214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/ts-0/wal/instance:
uuid: "023022f19f764ad792f6bb884b224151"
format_stamp: "Formatted at 2026-09-29 16:42:46 on dist-test-slave-l31f"
server_key: "ab459213a0c872b5b702cfde7d3f6abc"
server_key_iv: "977d53b3d20a7fd3b6a9bd8824b227ce"
server_key_version: "encryptionkey@0"
I20260929 16:42:46.670166 10214 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 16:42:46.673950 10228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:46.675539 10214 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 16:42:46.675743 10214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/ts-0/wal
uuid: "023022f19f764ad792f6bb884b224151"
format_stamp: "Formatted at 2026-09-29 16:42:46 on dist-test-slave-l31f"
server_key: "ab459213a0c872b5b702cfde7d3f6abc"
server_key_iv: "977d53b3d20a7fd3b6a9bd8824b227ce"
server_key_version: "encryptionkey@0"
I20260929 16:42:46.675997 10214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:42:46.702239 10214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:42:46.703269 10214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:42:46.703624 10214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:42:46.704638 10214 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:42:46.706377 10214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:42:46.706516 10214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:46.706662 10214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:42:46.706750 10214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:46.740537 10214 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.65:40331
I20260929 16:42:46.740595 10340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.65:40331 every 8 connection(s)
I20260929 16:42:46.742190 10214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
I20260929 16:42:46.751930 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 10214
I20260929 16:42:46.752153 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/ts-0/wal/instance
I20260929 16:42:46.752782 789 external_mini_cluster.cc:1428] Setting key 816fb8398ae2589f9d28e5f457154096
I20260929 16:42:46.755113 10343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:46.744897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:39105 (local address 127.0.197.126:45001)
0929 16:42:46.745382 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:46.745387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:46.746420 (+ 1033us) server_negotiation.cc:407] Connection header received
0929 16:42:46.748296 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:46.748302 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:46.748395 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:46.748516 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:46.749638 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:46.750381 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:46.752085 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:46.752548 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:46.752697 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:46.754234 (+ 1537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:46.754239 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:46.754243 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:42:46.754261 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:46.754299 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:42:46.754311 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:46.754413 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:46.754643 (+ 230us) server_negotiation.cc:299] Negotiation successful
0929 16:42:46.754741 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":291,"threads_started":1}
I20260929 16:42:46.755636 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.66:0
--local_ip_for_outbound_sockets=127.0.197.66
--webserver_interface=127.0.197.66
--webserver_port=0
--tserver_master_addrs=127.0.197.126:45001
--builtin_ntp_servers=127.0.197.84:46549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 16:42:46.756313 10344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:46.745371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:45001 (local address 127.0.197.65:39105)
0929 16:42:46.746265 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:46.746302 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:46.748006 (+ 1704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:46.748763 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:46.748772 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:46.748862 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:46.749457 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:46.749475 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:46.750548 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:46.750552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:46.751911 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:46.751926 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:46.752808 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:46.752815 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:46.753117 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:46.753876 (+ 759us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:42:46.753879 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:46.753923 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:42:46.754010 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:42:46.754412 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:42:46.754424 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:42:46.754434 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 16:42:46.754637 (+ 203us) client_negotiation.cc:240] Negotiation successful
0929 16:42:46.755140 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":270,"threads_started":1}
I20260929 16:42:46.758100 10341 heartbeater.cc:347] Connected to a master server at 127.0.197.126:45001
I20260929 16:42:46.758438 10341 heartbeater.cc:464] Registering TS with master...
I20260929 16:42:46.759326 10341 heartbeater.cc:511] Master 127.0.197.126:45001 requested a full tablet report, sending...
I20260929 16:42:46.761799 10155 ts_manager.cc:194] Registered new tserver with Master: 023022f19f764ad792f6bb884b224151 (127.0.197.65:40331)
I20260929 16:42:46.763758 10155 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.197.65:39105
W20260929 16:42:47.108134 10345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:42:47.108464 10345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:42:47.108619 10345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:42:47.116108 10345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:42:47.116233 10345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:42:47.116429 10345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.66
I20260929 16:42:47.125952 10345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:46549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.197.66
--webserver_port=0
--tserver_master_addrs=127.0.197.126:45001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.66
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:42:47.127362 10345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:42:47.129210 10345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:42:47.143576 10353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:47.143579 10351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:47.143587 10350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:42:47.144064 10345 server_base.cc:1034] running on GCE node
I20260929 16:42:47.144860 10345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:42:47.145905 10345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:42:47.147146 10345 hybrid_clock.cc:648] HybridClock initialized: now 1790700167147067 us; error 55 us; skew 500 ppm
I20260929 16:42:47.147498 10345 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:42:47.150017 10345 webserver.cc:460] Webserver started at http://127.0.197.66:33751/ using document root <none> and password file <none>
I20260929 16:42:47.150802 10345 fs_manager.cc:359] Metadata directory not provided
I20260929 16:42:47.150941 10345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:42:47.151315 10345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:42:47.154238 10345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/ts-1/data/instance:
uuid: "e41692d8af6e4980b8875c58354675ed"
format_stamp: "Formatted at 2026-09-29 16:42:47 on dist-test-slave-l31f"
server_key: "93af41980d987ef75543c8669bc8cef3"
server_key_iv: "a73dda70432a409b344e66bd0eebfd2a"
server_key_version: "encryptionkey@0"
I20260929 16:42:47.155114 10345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/ts-1/wal/instance:
uuid: "e41692d8af6e4980b8875c58354675ed"
format_stamp: "Formatted at 2026-09-29 16:42:47 on dist-test-slave-l31f"
server_key: "93af41980d987ef75543c8669bc8cef3"
server_key_iv: "a73dda70432a409b344e66bd0eebfd2a"
server_key_version: "encryptionkey@0"
I20260929 16:42:47.160495 10345 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 16:42:47.164315 10359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:47.165817 10345 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:42:47.166126 10345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/ts-1/wal
uuid: "e41692d8af6e4980b8875c58354675ed"
format_stamp: "Formatted at 2026-09-29 16:42:47 on dist-test-slave-l31f"
server_key: "93af41980d987ef75543c8669bc8cef3"
server_key_iv: "a73dda70432a409b344e66bd0eebfd2a"
server_key_version: "encryptionkey@0"
I20260929 16:42:47.166381 10345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:42:47.207825 10345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:42:47.208782 10345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:42:47.209105 10345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:42:47.210147 10345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:42:47.211854 10345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:42:47.211984 10345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:47.212126 10345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:42:47.212215 10345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:47.247875 10345 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.66:45013
I20260929 16:42:47.247934 10471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.66:45013 every 8 connection(s)
I20260929 16:42:47.249241 10345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
I20260929 16:42:47.252589 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 10345
I20260929 16:42:47.252723 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/ts-1/wal/instance
I20260929 16:42:47.253185 789 external_mini_cluster.cc:1428] Setting key b9856bb227b254dd7f69e24cb1e2e4d9
I20260929 16:42:47.255951 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.67:0
--local_ip_for_outbound_sockets=127.0.197.67
--webserver_interface=127.0.197.67
--webserver_port=0
--tserver_master_addrs=127.0.197.126:45001
--builtin_ntp_servers=127.0.197.84:46549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 16:42:47.264170 10343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:47.252073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:44303 (local address 127.0.197.126:45001)
0929 16:42:47.252253 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:47.252261 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:47.253841 (+ 1580us) server_negotiation.cc:407] Connection header received
0929 16:42:47.255893 (+ 2052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:47.255900 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:47.255996 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:47.256116 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:47.257461 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:47.258596 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:47.260414 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:47.261181 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:47.261330 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:47.263182 (+ 1852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:47.263191 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:47.263197 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:42:47.263224 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:47.263269 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:42:47.263288 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:47.263392 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:47.263659 (+ 267us) server_negotiation.cc:299] Negotiation successful
0929 16:42:47.263776 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 16:42:47.265232 10474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:47.252503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:45001 (local address 127.0.197.66:44303)
0929 16:42:47.253650 (+ 1147us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:47.253705 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:47.255605 (+ 1900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:47.256413 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:47.256422 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:47.256524 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:47.257273 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:47.257292 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:47.258805 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:47.258809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:47.260238 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:47.260252 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:47.261457 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:47.261466 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:47.261802 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:47.262701 (+ 899us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:42:47.262706 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:47.262773 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:42:47.262904 (+ 131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:42:47.263410 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:42:47.263419 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:42:47.263427 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 16:42:47.263595 (+ 168us) client_negotiation.cc:240] Negotiation successful
0929 16:42:47.263938 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":417,"threads_started":1}
I20260929 16:42:47.266948 10472 heartbeater.cc:347] Connected to a master server at 127.0.197.126:45001
I20260929 16:42:47.267333 10472 heartbeater.cc:464] Registering TS with master...
I20260929 16:42:47.268249 10472 heartbeater.cc:511] Master 127.0.197.126:45001 requested a full tablet report, sending...
I20260929 16:42:47.270030 10155 ts_manager.cc:194] Registered new tserver with Master: e41692d8af6e4980b8875c58354675ed (127.0.197.66:45013)
I20260929 16:42:47.270902 10155 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.197.66:44303
W20260929 16:42:47.604436 10475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:42:47.604758 10475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:42:47.604907 10475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:42:47.613152 10475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:42:47.613271 10475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:42:47.613458 10475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.67
I20260929 16:42:47.623229 10475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:46549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.197.67
--webserver_port=0
--tserver_master_addrs=127.0.197.126:45001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.67
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:42:47.624665 10475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:42:47.626550 10475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:42:47.641507 10480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:47.641472 10481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:47.641448 10483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:42:47.642422 10475 server_base.cc:1034] running on GCE node
I20260929 16:42:47.643254 10475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:42:47.644353 10475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:42:47.645634 10475 hybrid_clock.cc:648] HybridClock initialized: now 1790700167645564 us; error 69 us; skew 500 ppm
I20260929 16:42:47.646101 10475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:42:47.649091 10475 webserver.cc:460] Webserver started at http://127.0.197.67:39719/ using document root <none> and password file <none>
I20260929 16:42:47.649873 10475 fs_manager.cc:359] Metadata directory not provided
I20260929 16:42:47.650024 10475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:42:47.650362 10475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:42:47.653601 10475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/ts-2/data/instance:
uuid: "c280a8b0dc5241c1a012426913f0009e"
format_stamp: "Formatted at 2026-09-29 16:42:47 on dist-test-slave-l31f"
server_key: "e62208191df5dec019862e1f019ad31e"
server_key_iv: "95dc2eb685801b519b124daa08bc747d"
server_key_version: "encryptionkey@0"
I20260929 16:42:47.654426 10475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/ts-2/wal/instance:
uuid: "c280a8b0dc5241c1a012426913f0009e"
format_stamp: "Formatted at 2026-09-29 16:42:47 on dist-test-slave-l31f"
server_key: "e62208191df5dec019862e1f019ad31e"
server_key_iv: "95dc2eb685801b519b124daa08bc747d"
server_key_version: "encryptionkey@0"
I20260929 16:42:47.659829 10475 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 16:42:47.663585 10489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:47.665226 10475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 16:42:47.665439 10475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/ts-2/wal
uuid: "c280a8b0dc5241c1a012426913f0009e"
format_stamp: "Formatted at 2026-09-29 16:42:47 on dist-test-slave-l31f"
server_key: "e62208191df5dec019862e1f019ad31e"
server_key_iv: "95dc2eb685801b519b124daa08bc747d"
server_key_version: "encryptionkey@0"
I20260929 16:42:47.665676 10475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:42:47.685488 10475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:42:47.686604 10475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:42:47.686947 10475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:42:47.688115 10475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:42:47.689623 10475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:42:47.689715 10475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:47.689862 10475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:42:47.689953 10475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:47.726188 10475 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.67:39825
I20260929 16:42:47.726222 10601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.67:39825 every 8 connection(s)
I20260929 16:42:47.727741 10475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
I20260929 16:42:47.734854 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 10475
I20260929 16:42:47.734995 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790700068255611-789-0/minicluster-data/ts-2/wal/instance
I20260929 16:42:47.735527 789 external_mini_cluster.cc:1428] Setting key cc08223337dff4ea33ac04352bb0f934
I20260929 16:42:47.741783 10343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:47.730480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:54885 (local address 127.0.197.126:45001)
0929 16:42:47.730689 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:47.730694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:47.732186 (+ 1492us) server_negotiation.cc:407] Connection header received
0929 16:42:47.733867 (+ 1681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:47.733872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:47.733942 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:47.734042 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:47.735282 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:47.736200 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:47.738457 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:47.739074 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:47.739181 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:47.740761 (+ 1580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:47.740769 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:47.740777 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:42:47.740804 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:47.740843 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:42:47.740863 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:47.740996 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:47.741257 (+ 261us) server_negotiation.cc:299] Negotiation successful
0929 16:42:47.741370 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260929 16:42:47.742722 10604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:47.731095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:45001 (local address 127.0.197.67:54885)
0929 16:42:47.732003 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:47.732039 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:47.733586 (+ 1547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:47.734258 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:47.734266 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:47.734358 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:47.735025 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:47.735085 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:47.736374 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:47.736379 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:47.738289 (+ 1910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:47.738310 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:47.739329 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:47.739334 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:47.739582 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:47.740388 (+ 806us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:42:47.740391 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:47.740434 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:42:47.740521 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:42:47.741021 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:42:47.741030 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:42:47.741037 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 16:42:47.741216 (+ 179us) client_negotiation.cc:240] Negotiation successful
0929 16:42:47.741548 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":274,"threads_started":1}
I20260929 16:42:47.744321 10602 heartbeater.cc:347] Connected to a master server at 127.0.197.126:45001
I20260929 16:42:47.744638 10602 heartbeater.cc:464] Registering TS with master...
I20260929 16:42:47.745412 10602 heartbeater.cc:511] Master 127.0.197.126:45001 requested a full tablet report, sending...
I20260929 16:42:47.746960 10155 ts_manager.cc:194] Registered new tserver with Master: c280a8b0dc5241c1a012426913f0009e (127.0.197.67:39825)
I20260929 16:42:47.747821 10155 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.197.67:54885
I20260929 16:42:47.751761 789 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 16:42:47.758363 10343 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:42:47.756550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35702 (local address 127.0.197.126:45001)
0929 16:42:47.756851 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:47.756861 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:47.757441 (+ 580us) server_negotiation.cc:407] Connection header received
0929 16:42:47.757608 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:47.757612 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:47.757678 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:47.757763 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:47.758091 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35702: BlockingRecv error: recv got EOF from 127.0.0.1:35702 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 16:42:47.759315 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 10214
I20260929 16:42:47.767302 10341 heartbeater.cc:503] Master 127.0.197.126:45001 was elected leader, sending a full tablet report...
I20260929 16:42:47.768118 10611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:47.760319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48660 (local address 127.0.197.65:40331)
0929 16:42:47.760892 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:47.760899 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:47.760931 (+ 32us) server_negotiation.cc:407] Connection header received
0929 16:42:47.760994 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:47.761000 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:47.761213 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:47.761404 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:47.761833 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:47.762808 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:47.764174 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:47.764783 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:47.764951 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:47.766413 (+ 1462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:47.766446 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:47.766460 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:42:47.767171 (+ 711us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:47.767231 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:42:47.767261 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:47.767366 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:47.767582 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 16:42:47.767760 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":266,"threads_started":1}
I20260929 16:42:47.769549 10336 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:42:47.853572 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 10214
I20260929 16:42:47.870409 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 10345
I20260929 16:42:47.878649 10613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:47.871540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38666 (local address 127.0.197.66:45013)
0929 16:42:47.872175 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:47.872182 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:47.872211 (+ 29us) server_negotiation.cc:407] Connection header received
0929 16:42:47.872274 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:47.872280 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:47.872501 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:47.872641 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:47.872972 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:47.873971 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:47.875275 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:47.875848 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:47.875954 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:47.876943 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:47.876951 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:47.876964 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:42:47.877597 (+ 633us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:47.877646 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:42:47.877665 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:47.877766 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:47.877989 (+ 223us) server_negotiation.cc:299] Negotiation successful
0929 16:42:47.878239 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":333,"threads_started":1}
I20260929 16:42:47.879992 10467 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:42:47.963591 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 10345
I20260929 16:42:47.980192 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 10475
I20260929 16:42:47.988597 10615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:47.981195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33858 (local address 127.0.197.67:39825)
0929 16:42:47.981736 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:47.981744 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:47.981775 (+ 31us) server_negotiation.cc:407] Connection header received
0929 16:42:47.981835 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:47.981841 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:47.982063 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:47.982242 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:47.982702 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:47.983812 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:47.985153 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:47.985714 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:47.985878 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:47.987061 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:47.987078 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:47.987092 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:42:47.987735 (+ 643us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:47.987782 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:42:47.987807 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:47.987944 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:47.988094 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 16:42:47.988239 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":252,"threads_started":1}
I20260929 16:42:47.989955 10597 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:42:48.076296 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 10475
I20260929 16:42:48.093109 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 10124
I20260929 16:42:48.094400 10185 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:42:48.196750 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 10124
2026-09-29T16:42:48Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2640 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-29T16:42:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:42:48Z Disabled control of system clock
I20260929 16:42:48.275286 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:41653
--webserver_interface=127.0.197.126
--webserver_port=0
--builtin_ntp_servers=127.0.197.84:38443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.126:41653
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 16:42:48.622927 10621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:42:48.623373 10621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:42:48.623483 10621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:42:48.631325 10621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:42:48.631453 10621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:42:48.631490 10621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:42:48.631547 10621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:42:48.631592 10621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:42:48.641328 10621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:38443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.197.126:41653
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:41653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.197.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:42:48.642733 10621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:42:48.644675 10621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:42:48.657706 10629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:48.657796 10626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:48.657757 10627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:42:48.659577 10621 server_base.cc:1034] running on GCE node
I20260929 16:42:48.661000 10621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:42:48.662667 10621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:42:48.663918 10621 hybrid_clock.cc:648] HybridClock initialized: now 1790700168663826 us; error 102 us; skew 500 ppm
I20260929 16:42:48.664353 10621 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:42:48.666976 10621 webserver.cc:460] Webserver started at http://127.0.197.126:35397/ using document root <none> and password file <none>
I20260929 16:42:48.667857 10621 fs_manager.cc:359] Metadata directory not provided
I20260929 16:42:48.667954 10621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:42:48.668402 10621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:42:48.674068 10621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/master-0/data/instance:
uuid: "b6f775fbe97d427e8d79bce6938a28f6"
format_stamp: "Formatted at 2026-09-29 16:42:48 on dist-test-slave-l31f"
server_key: "562c00b2270d2541f44759b6587bdc2c"
server_key_iv: "13926012d19bafd0550e19a4805bbfe3"
server_key_version: "encryptionkey@0"
I20260929 16:42:48.675074 10621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/master-0/wal/instance:
uuid: "b6f775fbe97d427e8d79bce6938a28f6"
format_stamp: "Formatted at 2026-09-29 16:42:48 on dist-test-slave-l31f"
server_key: "562c00b2270d2541f44759b6587bdc2c"
server_key_iv: "13926012d19bafd0550e19a4805bbfe3"
server_key_version: "encryptionkey@0"
I20260929 16:42:48.681483 10621 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 16:42:48.685474 10635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:48.687140 10621 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 16:42:48.687357 10621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "b6f775fbe97d427e8d79bce6938a28f6"
format_stamp: "Formatted at 2026-09-29 16:42:48 on dist-test-slave-l31f"
server_key: "562c00b2270d2541f44759b6587bdc2c"
server_key_iv: "13926012d19bafd0550e19a4805bbfe3"
server_key_version: "encryptionkey@0"
I20260929 16:42:48.687731 10621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:42:48.695120 10621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:42:48.696058 10621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:42:48.696391 10621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:42:48.718708 10621 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.126:41653
I20260929 16:42:48.718753 10686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.126:41653 every 8 connection(s)
I20260929 16:42:48.720248 10621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:42:48.724447 10687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:42:48.730703 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 10621
I20260929 16:42:48.730969 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/master-0/wal/instance
I20260929 16:42:48.731645 789 external_mini_cluster.cc:1428] Setting key 7c062a980d270f6bde6d739c7251f606
I20260929 16:42:48.736358 10687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6f775fbe97d427e8d79bce6938a28f6: Bootstrap starting.
I20260929 16:42:48.740083 10687 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6f775fbe97d427e8d79bce6938a28f6: Neither blocks nor log segments found. Creating new log.
I20260929 16:42:48.740914 10690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:48.733833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35682 (local address 127.0.197.126:41653)
0929 16:42:48.734960 (+ 1127us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:48.734979 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:48.735090 (+ 111us) server_negotiation.cc:407] Connection header received
0929 16:42:48.736954 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:48.736978 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:48.737423 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:48.737874 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:48.739154 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:48.739160 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:48.739175 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:42:48.739190 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:48.739254 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:42:48.739271 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:48.739473 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:48.739622 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 16:42:48.739897 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":354,"threads_started":1}
I20260929 16:42:48.741356 10687 log.cc:824] T 00000000000000000000000000000000 P b6f775fbe97d427e8d79bce6938a28f6: Log is configured to *not* fsync() on all Append() calls
I20260929 16:42:48.744903 10687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6f775fbe97d427e8d79bce6938a28f6: No bootstrap required, opened a new log
I20260929 16:42:48.752633 10687 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6f775fbe97d427e8d79bce6938a28f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6f775fbe97d427e8d79bce6938a28f6" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 41653 } }
I20260929 16:42:48.753036 10687 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6f775fbe97d427e8d79bce6938a28f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:42:48.753104 10687 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6f775fbe97d427e8d79bce6938a28f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6f775fbe97d427e8d79bce6938a28f6, State: Initialized, Role: FOLLOWER
I20260929 16:42:48.753863 10687 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6f775fbe97d427e8d79bce6938a28f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6f775fbe97d427e8d79bce6938a28f6" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 41653 } }
I20260929 16:42:48.754148 10687 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6f775fbe97d427e8d79bce6938a28f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:42:48.754313 10687 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6f775fbe97d427e8d79bce6938a28f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:42:48.754510 10687 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6f775fbe97d427e8d79bce6938a28f6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:42:48.756891 10687 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6f775fbe97d427e8d79bce6938a28f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6f775fbe97d427e8d79bce6938a28f6" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 41653 } }
I20260929 16:42:48.757496 10687 leader_election.cc:302] T 00000000000000000000000000000000 P b6f775fbe97d427e8d79bce6938a28f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6f775fbe97d427e8d79bce6938a28f6; no voters:
I20260929 16:42:48.758456 10687 leader_election.cc:288] T 00000000000000000000000000000000 P b6f775fbe97d427e8d79bce6938a28f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:42:48.758661 10692 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6f775fbe97d427e8d79bce6938a28f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:42:48.759562 10692 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6f775fbe97d427e8d79bce6938a28f6 [term 1 LEADER]: Becoming Leader. State: Replica: b6f775fbe97d427e8d79bce6938a28f6, State: Running, Role: LEADER
I20260929 16:42:48.760109 10692 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6f775fbe97d427e8d79bce6938a28f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6f775fbe97d427e8d79bce6938a28f6" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 41653 } }
I20260929 16:42:48.761252 10687 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6f775fbe97d427e8d79bce6938a28f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:42:48.762969 10693 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6f775fbe97d427e8d79bce6938a28f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6f775fbe97d427e8d79bce6938a28f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6f775fbe97d427e8d79bce6938a28f6" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 41653 } } }
I20260929 16:42:48.763489 10693 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6f775fbe97d427e8d79bce6938a28f6 [sys.catalog]: This master's current role is: LEADER
I20260929 16:42:48.764096 10694 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6f775fbe97d427e8d79bce6938a28f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6f775fbe97d427e8d79bce6938a28f6. Latest consensus state: current_term: 1 leader_uuid: "b6f775fbe97d427e8d79bce6938a28f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6f775fbe97d427e8d79bce6938a28f6" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 41653 } } }
I20260929 16:42:48.764470 10694 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6f775fbe97d427e8d79bce6938a28f6 [sys.catalog]: This master's current role is: LEADER
I20260929 16:42:48.765931 10700 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:42:48.772334 10700 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:42:48.782229 10700 catalog_manager.cc:1347] Generated new cluster ID: 0578688994c64a8e8ca11b7022651228
I20260929 16:42:48.782370 10700 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:42:48.816941 10700 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:42:48.817457 10700 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:42:48.833285 10700 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6f775fbe97d427e8d79bce6938a28f6: Generated new TSK 0
I20260929 16:42:48.834255 10700 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:42:48.844719 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.65:0
--local_ip_for_outbound_sockets=127.0.197.65
--webserver_interface=127.0.197.65
--webserver_port=0
--tserver_master_addrs=127.0.197.126:41653
--builtin_ntp_servers=127.0.197.84:38443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 16:42:49.200197 10711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:42:49.200498 10711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:42:49.200654 10711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:42:49.208316 10711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:42:49.208439 10711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:42:49.208632 10711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.65
I20260929 16:42:49.218204 10711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:38443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.197.65
--webserver_port=0
--tserver_master_addrs=127.0.197.126:41653
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.65
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:42:49.219667 10711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:42:49.221557 10711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:42:49.235863 10719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:49.235824 10717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:49.235838 10716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:42:49.236546 10711 server_base.cc:1034] running on GCE node
I20260929 16:42:49.237282 10711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:42:49.238293 10711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:42:49.239530 10711 hybrid_clock.cc:648] HybridClock initialized: now 1790700169239473 us; error 45 us; skew 500 ppm
I20260929 16:42:49.239915 10711 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:42:49.242480 10711 webserver.cc:460] Webserver started at http://127.0.197.65:35505/ using document root <none> and password file <none>
I20260929 16:42:49.243464 10711 fs_manager.cc:359] Metadata directory not provided
I20260929 16:42:49.243611 10711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:42:49.243970 10711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:42:49.247097 10711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/ts-0/data/instance:
uuid: "12c4c066fb2c4081b6d323f9a69a21c7"
format_stamp: "Formatted at 2026-09-29 16:42:49 on dist-test-slave-l31f"
server_key: "9bcf8714124b8992120a323b599266b6"
server_key_iv: "c600fd29d767a01afcaf4ce5e889b50d"
server_key_version: "encryptionkey@0"
I20260929 16:42:49.247982 10711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/ts-0/wal/instance:
uuid: "12c4c066fb2c4081b6d323f9a69a21c7"
format_stamp: "Formatted at 2026-09-29 16:42:49 on dist-test-slave-l31f"
server_key: "9bcf8714124b8992120a323b599266b6"
server_key_iv: "c600fd29d767a01afcaf4ce5e889b50d"
server_key_version: "encryptionkey@0"
I20260929 16:42:49.253896 10711 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 16:42:49.257963 10725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:49.259745 10711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:42:49.259970 10711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/ts-0/wal
uuid: "12c4c066fb2c4081b6d323f9a69a21c7"
format_stamp: "Formatted at 2026-09-29 16:42:49 on dist-test-slave-l31f"
server_key: "9bcf8714124b8992120a323b599266b6"
server_key_iv: "c600fd29d767a01afcaf4ce5e889b50d"
server_key_version: "encryptionkey@0"
I20260929 16:42:49.260249 10711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:42:49.266788 10711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:42:49.267768 10711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:42:49.268066 10711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:42:49.269032 10711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:42:49.270612 10711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:42:49.270821 10711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:49.271091 10711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:42:49.271205 10711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260929 16:42:49.306497 10711 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.65:44069
I20260929 16:42:49.306613 10837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.65:44069 every 8 connection(s)
I20260929 16:42:49.307953 10711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
I20260929 16:42:49.309017 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 10711
I20260929 16:42:49.309222 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/ts-0/wal/instance
I20260929 16:42:49.309842 789 external_mini_cluster.cc:1428] Setting key b1e5ad3e3861a3b83820181173b84c9c
I20260929 16:42:49.313642 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.66:0
--local_ip_for_outbound_sockets=127.0.197.66
--webserver_interface=127.0.197.66
--webserver_port=0
--tserver_master_addrs=127.0.197.126:41653
--builtin_ntp_servers=127.0.197.84:38443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 16:42:49.319242 10840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:49.311425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:36741 (local address 127.0.197.126:41653)
0929 16:42:49.312183 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:49.312189 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:49.313477 (+ 1288us) server_negotiation.cc:407] Connection header received
0929 16:42:49.316523 (+ 3046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:49.316528 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:49.316590 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:49.316694 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:49.318419 (+ 1725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:49.318423 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:49.318427 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:42:49.318446 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:49.318496 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:42:49.318509 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:49.318616 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:49.318862 (+ 246us) server_negotiation.cc:299] Negotiation successful
0929 16:42:49.318920 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":498,"threads_started":1}
I20260929 16:42:49.320643 10841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:49.311939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:41653 (local address 127.0.197.65:36741)
0929 16:42:49.313292 (+ 1353us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:49.313344 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:49.316081 (+ 2737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:49.317021 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:49.317036 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:49.317196 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:49.317982 (+ 786us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:42:49.317987 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:49.318044 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:42:49.318187 (+ 143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:42:49.318620 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:42:49.318634 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:42:49.318644 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 16:42:49.318942 (+ 298us) client_negotiation.cc:240] Negotiation successful
0929 16:42:49.319289 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"thread_start_us":372,"threads_started":1}
I20260929 16:42:49.322894 10838 heartbeater.cc:347] Connected to a master server at 127.0.197.126:41653
I20260929 16:42:49.323298 10838 heartbeater.cc:464] Registering TS with master...
I20260929 16:42:49.323902 10838 heartbeater.cc:511] Master 127.0.197.126:41653 requested a full tablet report, sending...
I20260929 16:42:49.326195 10652 ts_manager.cc:194] Registered new tserver with Master: 12c4c066fb2c4081b6d323f9a69a21c7 (127.0.197.65:44069)
W20260929 16:42:49.659454 10842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:42:49.659765 10842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:42:49.659857 10842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:42:49.667891 10842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:42:49.668041 10842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:42:49.668187 10842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.66
I20260929 16:42:49.677719 10842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:38443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.197.66
--webserver_port=0
--tserver_master_addrs=127.0.197.126:41653
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.66
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:42:49.678983 10842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:42:49.680750 10842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:42:49.694231 10847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:49.694257 10850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:49.694197 10848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:42:49.695374 10842 server_base.cc:1034] running on GCE node
I20260929 16:42:49.696094 10842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:42:49.697095 10842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:42:49.698297 10842 hybrid_clock.cc:648] HybridClock initialized: now 1790700169698259 us; error 36 us; skew 500 ppm
I20260929 16:42:49.698637 10842 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:42:49.701246 10842 webserver.cc:460] Webserver started at http://127.0.197.66:42655/ using document root <none> and password file <none>
I20260929 16:42:49.701984 10842 fs_manager.cc:359] Metadata directory not provided
I20260929 16:42:49.702085 10842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:42:49.702370 10842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:42:49.705386 10842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/ts-1/data/instance:
uuid: "6e3c81ce527e401c9a38a15c82f90c5f"
format_stamp: "Formatted at 2026-09-29 16:42:49 on dist-test-slave-l31f"
server_key: "831500f55fb82ddb7f604fcb29a0b1ff"
server_key_iv: "a0f488ee720ac3e980bf85237695f2e9"
server_key_version: "encryptionkey@0"
I20260929 16:42:49.706182 10842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/ts-1/wal/instance:
uuid: "6e3c81ce527e401c9a38a15c82f90c5f"
format_stamp: "Formatted at 2026-09-29 16:42:49 on dist-test-slave-l31f"
server_key: "831500f55fb82ddb7f604fcb29a0b1ff"
server_key_iv: "a0f488ee720ac3e980bf85237695f2e9"
server_key_version: "encryptionkey@0"
I20260929 16:42:49.712136 10842 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 16:42:49.716352 10856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:49.717970 10842 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260929 16:42:49.718170 10842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/ts-1/wal
uuid: "6e3c81ce527e401c9a38a15c82f90c5f"
format_stamp: "Formatted at 2026-09-29 16:42:49 on dist-test-slave-l31f"
server_key: "831500f55fb82ddb7f604fcb29a0b1ff"
server_key_iv: "a0f488ee720ac3e980bf85237695f2e9"
server_key_version: "encryptionkey@0"
I20260929 16:42:49.718441 10842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:42:49.724471 10842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:42:49.725327 10842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:42:49.725630 10842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:42:49.726613 10842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:42:49.728256 10842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:42:49.728350 10842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:49.728511 10842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:42:49.728601 10842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:49.763571 10842 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.66:40599
I20260929 16:42:49.763612 10968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.66:40599 every 8 connection(s)
I20260929 16:42:49.765065 10842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
I20260929 16:42:49.773443 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 10842
I20260929 16:42:49.773710 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/ts-1/wal/instance
I20260929 16:42:49.774297 789 external_mini_cluster.cc:1428] Setting key a93f2adf759207f1554a65e1038a9bd5
I20260929 16:42:49.774631 10840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:49.768038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:44175 (local address 127.0.197.126:41653)
0929 16:42:49.768264 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:49.768270 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:49.769902 (+ 1632us) server_negotiation.cc:407] Connection header received
0929 16:42:49.771924 (+ 2022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:49.771931 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:49.772014 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:49.772129 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:49.773777 (+ 1648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:49.773787 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:49.773794 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:42:49.773818 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:49.773860 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:42:49.773880 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:49.774032 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:49.774280 (+ 248us) server_negotiation.cc:299] Negotiation successful
0929 16:42:49.774350 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 16:42:49.775842 10971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:49.768566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:41653 (local address 127.0.197.66:44175)
0929 16:42:49.769606 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:49.769736 (+ 130us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:49.771451 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:49.772350 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:49.772359 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:49.772462 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:49.773317 (+ 855us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:42:49.773320 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:49.773374 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:42:49.773539 (+ 165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:42:49.774052 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:42:49.774062 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:42:49.774069 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 16:42:49.774321 (+ 252us) client_negotiation.cc:240] Negotiation successful
0929 16:42:49.774674 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":278,"threads_started":1}
I20260929 16:42:49.777391 10969 heartbeater.cc:347] Connected to a master server at 127.0.197.126:41653
I20260929 16:42:49.777510 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.67:0
--local_ip_for_outbound_sockets=127.0.197.67
--webserver_interface=127.0.197.67
--webserver_port=0
--tserver_master_addrs=127.0.197.126:41653
--builtin_ntp_servers=127.0.197.84:38443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 16:42:49.777748 10969 heartbeater.cc:464] Registering TS with master...
I20260929 16:42:49.778380 10969 heartbeater.cc:511] Master 127.0.197.126:41653 requested a full tablet report, sending...
I20260929 16:42:49.780057 10652 ts_manager.cc:194] Registered new tserver with Master: 6e3c81ce527e401c9a38a15c82f90c5f (127.0.197.66:40599)
W20260929 16:42:50.130918 10972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:42:50.131309 10972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:42:50.131464 10972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:42:50.139089 10972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:42:50.139227 10972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:42:50.139431 10972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.67
I20260929 16:42:50.149765 10972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:38443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.197.67
--webserver_port=0
--tserver_master_addrs=127.0.197.126:41653
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.67
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:42:50.151332 10972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:42:50.153187 10972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:42:50.167227 10977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:50.167274 10980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:50.167222 10978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:42:50.167996 10972 server_base.cc:1034] running on GCE node
I20260929 16:42:50.168663 10972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:42:50.169723 10972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:42:50.170949 10972 hybrid_clock.cc:648] HybridClock initialized: now 1790700170170909 us; error 65 us; skew 500 ppm
I20260929 16:42:50.171365 10972 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:42:50.173977 10972 webserver.cc:460] Webserver started at http://127.0.197.67:37823/ using document root <none> and password file <none>
I20260929 16:42:50.174736 10972 fs_manager.cc:359] Metadata directory not provided
I20260929 16:42:50.174882 10972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:42:50.175274 10972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:42:50.178335 10972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/ts-2/data/instance:
uuid: "d859ff5e20364b7bbef48824a276b410"
format_stamp: "Formatted at 2026-09-29 16:42:50 on dist-test-slave-l31f"
server_key: "dc938ab962b2a50d108660edf475f2c6"
server_key_iv: "ff6b1994edd02c9323c14c46e32f5198"
server_key_version: "encryptionkey@0"
I20260929 16:42:50.179391 10972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/ts-2/wal/instance:
uuid: "d859ff5e20364b7bbef48824a276b410"
format_stamp: "Formatted at 2026-09-29 16:42:50 on dist-test-slave-l31f"
server_key: "dc938ab962b2a50d108660edf475f2c6"
server_key_iv: "ff6b1994edd02c9323c14c46e32f5198"
server_key_version: "encryptionkey@0"
I20260929 16:42:50.185391 10972 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 16:42:50.189378 10986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:50.190878 10972 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:42:50.191126 10972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/ts-2/wal
uuid: "d859ff5e20364b7bbef48824a276b410"
format_stamp: "Formatted at 2026-09-29 16:42:50 on dist-test-slave-l31f"
server_key: "dc938ab962b2a50d108660edf475f2c6"
server_key_iv: "ff6b1994edd02c9323c14c46e32f5198"
server_key_version: "encryptionkey@0"
I20260929 16:42:50.191562 10972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:42:50.197860 10972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:42:50.198762 10972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:42:50.199085 10972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:42:50.200074 10972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:42:50.201584 10972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:42:50.201707 10972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:50.201844 10972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:42:50.201936 10972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:50.237118 10972 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.67:43519
I20260929 16:42:50.237180 11098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.67:43519 every 8 connection(s)
I20260929 16:42:50.238536 10972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
I20260929 16:42:50.242913 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 10972
I20260929 16:42:50.243153 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790700068255611-789-0/minicluster-data/ts-2/wal/instance
I20260929 16:42:50.243733 789 external_mini_cluster.cc:1428] Setting key f6b9a09348988f273aac4ac7de5fd8ec
I20260929 16:42:50.247361 10840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:50.241144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:54803 (local address 127.0.197.126:41653)
0929 16:42:50.241329 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:50.241333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:50.242719 (+ 1386us) server_negotiation.cc:407] Connection header received
0929 16:42:50.244985 (+ 2266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:50.244991 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:50.245080 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:50.245207 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:50.246583 (+ 1376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:50.246591 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:50.246597 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:42:50.246615 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:50.246658 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:42:50.246670 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:50.246799 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:50.246996 (+ 197us) server_negotiation.cc:299] Negotiation successful
0929 16:42:50.247101 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 16:42:50.248404 11101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:50.241573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:41653 (local address 127.0.197.67:54803)
0929 16:42:50.242484 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:50.242571 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:50.244699 (+ 2128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:50.245455 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:50.245464 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:50.245571 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:50.246285 (+ 714us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:42:50.246288 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:50.246328 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:42:50.246413 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:42:50.246768 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:42:50.246780 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:42:50.246789 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 16:42:50.247111 (+ 322us) client_negotiation.cc:240] Negotiation successful
0929 16:42:50.247405 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":294,"threads_started":1}
I20260929 16:42:50.250375 11099 heartbeater.cc:347] Connected to a master server at 127.0.197.126:41653
I20260929 16:42:50.250723 11099 heartbeater.cc:464] Registering TS with master...
I20260929 16:42:50.251416 11099 heartbeater.cc:511] Master 127.0.197.126:41653 requested a full tablet report, sending...
I20260929 16:42:50.253108 10652 ts_manager.cc:194] Registered new tserver with Master: d859ff5e20364b7bbef48824a276b410 (127.0.197.67:43519)
I20260929 16:42:50.259389 789 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 16:42:50.266124 10840 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:42:50.264314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35692 (local address 127.0.197.126:41653)
0929 16:42:50.264487 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:50.264494 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:50.265037 (+ 543us) server_negotiation.cc:407] Connection header received
0929 16:42:50.265291 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:50.265299 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:50.265368 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:50.265478 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:50.265829 (+ 351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35692: BlockingRecv error: recv got EOF from 127.0.0.1:35692 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 16:42:50.266748 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 10711
I20260929 16:42:50.270550 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:50.267779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37386 (local address 127.0.197.65:44069)
0929 16:42:50.268303 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:50.268310 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:50.268337 (+ 27us) server_negotiation.cc:407] Connection header received
0929 16:42:50.268395 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:50.268401 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:50.268618 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:50.268790 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:50.269815 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:50.269821 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:50.269833 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:42:50.269850 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:50.269888 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:42:50.269901 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:50.270063 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:50.270143 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 16:42:50.270243 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":228,"threads_started":1}
I20260929 16:42:50.271968 10833 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:42:50.352536 10838 heartbeater.cc:503] Master 127.0.197.126:41653 was elected leader, sending a full tablet report...
I20260929 16:42:50.354355 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 10711
I20260929 16:42:50.371443 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 10842
I20260929 16:42:50.375413 11110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:50.372489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33680 (local address 127.0.197.66:40599)
0929 16:42:50.372980 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:50.372987 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:50.373014 (+ 27us) server_negotiation.cc:407] Connection header received
0929 16:42:50.373078 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:50.373084 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:50.373304 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:50.373436 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:50.374475 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:50.374482 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:50.374494 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:42:50.374513 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:50.374577 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:42:50.374593 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:50.374705 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:50.374891 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 16:42:50.375016 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":226,"threads_started":1}
I20260929 16:42:50.376773 10964 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:42:50.461138 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 10842
I20260929 16:42:50.478127 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 10972
I20260929 16:42:50.482312 11112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:50.479232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51244 (local address 127.0.197.67:43519)
0929 16:42:50.479882 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:50.479889 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:50.479920 (+ 31us) server_negotiation.cc:407] Connection header received
0929 16:42:50.479981 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:50.479988 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:50.480219 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:50.480403 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:50.481571 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:50.481578 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:50.481589 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:42:50.481605 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:50.481643 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:42:50.481656 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:50.481846 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:50.481919 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 16:42:50.482020 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":348,"threads_started":1}
I20260929 16:42:50.483857 11094 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:42:50.566931 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 10972
I20260929 16:42:50.583992 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 10621
I20260929 16:42:50.585652 10682 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:42:50.687395 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 10621
2026-09-29T16:42:50Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2490 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:42:50 dist-test-slave-l31f krb5kdc[11117](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:42:50 dist-test-slave-l31f krb5kdc[11117](info): set up 2 sockets
Sep 29 16:42:50 dist-test-slave-l31f krb5kdc[11117](info): commencing operation
krb5kdc: starting...
W20260929 16:42:51.820433 789 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/krb5kdc/test-user.keytab.
Sep 29 16:42:51 dist-test-slave-l31f krb5kdc[11117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:42:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:42:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.197.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
I20260929 16:42:52.083716 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:33313
--webserver_interface=127.0.197.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:38647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.126:33313
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:42:52.434494 11130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:42:52.434832 11130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:42:52.434976 11130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:42:52.442811 11130 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:42:52.442955 11130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:42:52.443065 11130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:42:52.443146 11130 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:42:52.443209 11130 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:42:52.453956 11130 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:38647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.197.126:33313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:33313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.197.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:42:52.455509 11130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:42:52.456971 11130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:42:52.469607 11136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:52.469683 11135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:52.469642 11138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:42:52.470665 11130 server_base.cc:1034] running on GCE node
I20260929 16:42:52.471582 11130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:42:52.473102 11130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:42:52.474334 11130 hybrid_clock.cc:648] HybridClock initialized: now 1790700172474263 us; error 70 us; skew 500 ppm
I20260929 16:42:52.474735 11130 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:42:52 dist-test-slave-l31f krb5kdc[11117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:42:52.484463 11130 init.cc:377] Logged in from keytab as kudu/127.0.197.126@KRBTEST.COM (short username kudu)
I20260929 16:42:52.486320 11130 webserver.cc:460] Webserver started at http://127.0.197.126:40827/ using document root <none> and password file <none>
I20260929 16:42:52.487181 11130 fs_manager.cc:359] Metadata directory not provided
I20260929 16:42:52.487347 11130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:42:52.487687 11130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:42:52.490958 11130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/master-0/data/instance:
uuid: "8ee2768f3c2b42bc8dbfc53d4ee2bed6"
format_stamp: "Formatted at 2026-09-29 16:42:52 on dist-test-slave-l31f"
server_key: "54800419e3a0940b973884e629bec607"
server_key_iv: "8ee391af60eefe40c212ae753e29833b"
server_key_version: "encryptionkey@0"
I20260929 16:42:52.491888 11130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/master-0/wal/instance:
uuid: "8ee2768f3c2b42bc8dbfc53d4ee2bed6"
format_stamp: "Formatted at 2026-09-29 16:42:52 on dist-test-slave-l31f"
server_key: "54800419e3a0940b973884e629bec607"
server_key_iv: "8ee391af60eefe40c212ae753e29833b"
server_key_version: "encryptionkey@0"
I20260929 16:42:52.497956 11130 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260929 16:42:52.501889 11145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:52.503700 11130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:42:52.503926 11130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "8ee2768f3c2b42bc8dbfc53d4ee2bed6"
format_stamp: "Formatted at 2026-09-29 16:42:52 on dist-test-slave-l31f"
server_key: "54800419e3a0940b973884e629bec607"
server_key_iv: "8ee391af60eefe40c212ae753e29833b"
server_key_version: "encryptionkey@0"
I20260929 16:42:52.504153 11130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:42:52.522362 11130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:42:52.527573 11130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:42:52.527904 11130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:42:52.548727 11130 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.126:33313
I20260929 16:42:52.548758 11196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.126:33313 every 8 connection(s)
I20260929 16:42:52.550251 11130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:42:52.554387 11197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:42:52.558966 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 11130
I20260929 16:42:52.559266 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/master-0/wal/instance
I20260929 16:42:52.560062 789 external_mini_cluster.cc:1428] Setting key 7eaa2e33c98abe21bd12aecc0394ec2d
I20260929 16:42:52.569978 11197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ee2768f3c2b42bc8dbfc53d4ee2bed6: Bootstrap starting.
I20260929 16:42:52.574206 11197 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ee2768f3c2b42bc8dbfc53d4ee2bed6: Neither blocks nor log segments found. Creating new log.
I20260929 16:42:52.575754 11197 log.cc:824] T 00000000000000000000000000000000 P 8ee2768f3c2b42bc8dbfc53d4ee2bed6: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:42:52 dist-test-slave-l31f krb5kdc[11117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:42:52.579744 11197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ee2768f3c2b42bc8dbfc53d4ee2bed6: No bootstrap required, opened a new log
I20260929 16:42:52.587215 11200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:52.562456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55194 (local address 127.0.197.126:33313)
0929 16:42:52.563420 (+ 964us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:52.563434 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:52.563481 (+ 47us) server_negotiation.cc:407] Connection header received
0929 16:42:52.565182 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:52.565210 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:52.565712 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:52.566105 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:52.568243 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:52.569688 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:52.571634 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:52.572926 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:52.573102 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:52.578500 (+ 5398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:52.578518 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:52.578564 (+ 46us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:52.578610 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:52.582245 (+ 3635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:52.583251 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:52.583259 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:52.583266 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:52.583528 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:52.583972 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:52.583977 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:52.583980 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:52.584704 (+ 724us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:52.584982 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:52.585512 (+ 530us) server_negotiation.cc:299] Negotiation successful
0929 16:42:52.586022 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":418,"threads_started":1}
I20260929 16:42:52.588984 11197 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ee2768f3c2b42bc8dbfc53d4ee2bed6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee2768f3c2b42bc8dbfc53d4ee2bed6" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33313 } }
I20260929 16:42:52.589376 11197 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ee2768f3c2b42bc8dbfc53d4ee2bed6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:42:52.589515 11197 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ee2768f3c2b42bc8dbfc53d4ee2bed6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee2768f3c2b42bc8dbfc53d4ee2bed6, State: Initialized, Role: FOLLOWER
I20260929 16:42:52.590168 11197 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ee2768f3c2b42bc8dbfc53d4ee2bed6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee2768f3c2b42bc8dbfc53d4ee2bed6" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33313 } }
I20260929 16:42:52.590452 11197 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ee2768f3c2b42bc8dbfc53d4ee2bed6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:42:52.590602 11197 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ee2768f3c2b42bc8dbfc53d4ee2bed6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:42:52.590790 11197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ee2768f3c2b42bc8dbfc53d4ee2bed6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:42:52.593039 11197 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ee2768f3c2b42bc8dbfc53d4ee2bed6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee2768f3c2b42bc8dbfc53d4ee2bed6" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33313 } }
I20260929 16:42:52.593614 11197 leader_election.cc:302] T 00000000000000000000000000000000 P 8ee2768f3c2b42bc8dbfc53d4ee2bed6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ee2768f3c2b42bc8dbfc53d4ee2bed6; no voters:
I20260929 16:42:52.594269 11197 leader_election.cc:288] T 00000000000000000000000000000000 P 8ee2768f3c2b42bc8dbfc53d4ee2bed6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:42:52.594472 11202 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ee2768f3c2b42bc8dbfc53d4ee2bed6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:42:52.595711 11202 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ee2768f3c2b42bc8dbfc53d4ee2bed6 [term 1 LEADER]: Becoming Leader. State: Replica: 8ee2768f3c2b42bc8dbfc53d4ee2bed6, State: Running, Role: LEADER
I20260929 16:42:52.596287 11202 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ee2768f3c2b42bc8dbfc53d4ee2bed6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee2768f3c2b42bc8dbfc53d4ee2bed6" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33313 } }
I20260929 16:42:52.596724 11197 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ee2768f3c2b42bc8dbfc53d4ee2bed6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:42:52.599431 11204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ee2768f3c2b42bc8dbfc53d4ee2bed6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ee2768f3c2b42bc8dbfc53d4ee2bed6. Latest consensus state: current_term: 1 leader_uuid: "8ee2768f3c2b42bc8dbfc53d4ee2bed6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee2768f3c2b42bc8dbfc53d4ee2bed6" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33313 } } }
I20260929 16:42:52.599865 11204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ee2768f3c2b42bc8dbfc53d4ee2bed6 [sys.catalog]: This master's current role is: LEADER
I20260929 16:42:52.599519 11203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ee2768f3c2b42bc8dbfc53d4ee2bed6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ee2768f3c2b42bc8dbfc53d4ee2bed6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee2768f3c2b42bc8dbfc53d4ee2bed6" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33313 } } }
I20260929 16:42:52.601835 11203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ee2768f3c2b42bc8dbfc53d4ee2bed6 [sys.catalog]: This master's current role is: LEADER
I20260929 16:42:52.610225 11208 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:42:52.616406 11208 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:42:52.626765 11208 catalog_manager.cc:1347] Generated new cluster ID: 37c1c96931734ca78872d1971f09dd6e
I20260929 16:42:52.626878 11208 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:42:52.643585 11208 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:42:52.645501 11208 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:42:52.656014 11208 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ee2768f3c2b42bc8dbfc53d4ee2bed6: Generated new TSK 0
I20260929 16:42:52.657290 11208 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.197.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab.
Entry for principal kudu/127.0.197.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab.
I20260929 16:42:52.738863 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.65:0
--local_ip_for_outbound_sockets=127.0.197.65
--webserver_interface=127.0.197.65
--webserver_port=0
--tserver_master_addrs=127.0.197.126:33313
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab
--principal=kudu/127.0.197.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:38647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:42:53.095395 11224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:42:53.095768 11224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:42:53.095923 11224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:42:53.103595 11224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:42:53.103731 11224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:42:53.103917 11224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.65
I20260929 16:42:53.114877 11224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:38647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab
--principal=kudu/127.0.197.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.197.65
--webserver_port=0
--tserver_master_addrs=127.0.197.126:33313
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.65
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:42:53.116406 11224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:42:53.117949 11224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:42:53.132642 11229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:53.132715 11232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:53.132733 11230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:42:53.133723 11224 server_base.cc:1034] running on GCE node
I20260929 16:42:53.134625 11224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:42:53.135738 11224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:42:53.136968 11224 hybrid_clock.cc:648] HybridClock initialized: now 1790700173136911 us; error 52 us; skew 500 ppm
I20260929 16:42:53.137274 11224 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:42:53 dist-test-slave-l31f krb5kdc[11117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:42:53.140910 11224 init.cc:377] Logged in from keytab as kudu/127.0.197.65@KRBTEST.COM (short username kudu)
I20260929 16:42:53.142544 11224 webserver.cc:460] Webserver started at http://127.0.197.65:32813/ using document root <none> and password file <none>
I20260929 16:42:53.143415 11224 fs_manager.cc:359] Metadata directory not provided
I20260929 16:42:53.143558 11224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:42:53.143891 11224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:42:53.146930 11224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/ts-0/data/instance:
uuid: "69508395076f45daa3148415ef9b52d5"
format_stamp: "Formatted at 2026-09-29 16:42:53 on dist-test-slave-l31f"
server_key: "c0b42d072f0650097fea84f6a35bb87c"
server_key_iv: "88dfd2012eff9995fa1e185b4ca9e231"
server_key_version: "encryptionkey@0"
I20260929 16:42:53.147846 11224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/ts-0/wal/instance:
uuid: "69508395076f45daa3148415ef9b52d5"
format_stamp: "Formatted at 2026-09-29 16:42:53 on dist-test-slave-l31f"
server_key: "c0b42d072f0650097fea84f6a35bb87c"
server_key_iv: "88dfd2012eff9995fa1e185b4ca9e231"
server_key_version: "encryptionkey@0"
I20260929 16:42:53.153555 11224 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:42:53.157779 11239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:53.159560 11224 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 16:42:53.159770 11224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/ts-0/wal
uuid: "69508395076f45daa3148415ef9b52d5"
format_stamp: "Formatted at 2026-09-29 16:42:53 on dist-test-slave-l31f"
server_key: "c0b42d072f0650097fea84f6a35bb87c"
server_key_iv: "88dfd2012eff9995fa1e185b4ca9e231"
server_key_version: "encryptionkey@0"
I20260929 16:42:53.159974 11224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:42:53.173714 11224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:42:53.179164 11224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:42:53.179520 11224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:42:53.180637 11224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:42:53.182067 11224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:42:53.182145 11224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:53.182301 11224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:42:53.182394 11224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:53.219427 11224 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.65:39085
I20260929 16:42:53.219470 11351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.65:39085 every 8 connection(s)
I20260929 16:42:53.220808 11224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
I20260929 16:42:53.228446 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 11224
I20260929 16:42:53.228652 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/ts-0/wal/instance
I20260929 16:42:53.229333 789 external_mini_cluster.cc:1428] Setting key ea9e072d052c7a2355c0aedc89719256
Sep 29 16:42:53 dist-test-slave-l31f krb5kdc[11117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.65@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:42:53.242523 11354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:53.223700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:57701 (local address 127.0.197.126:33313)
0929 16:42:53.224175 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:53.224181 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:53.225093 (+ 912us) server_negotiation.cc:407] Connection header received
0929 16:42:53.227179 (+ 2086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:53.227186 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:53.227280 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:53.227406 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:53.229210 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:53.229981 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:53.231743 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:53.232138 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:53.232280 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:53.236441 (+ 4161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:53.236447 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:53.236450 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:53.236485 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:53.239231 (+ 2746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:53.240083 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:53.240088 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:53.240090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:53.240218 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:53.240685 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:53.240697 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:53.240700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:53.240971 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:53.241129 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:53.241894 (+ 765us) server_negotiation.cc:299] Negotiation successful
0929 16:42:53.242133 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":263,"threads_started":1}
I20260929 16:42:53.243630 11355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:53.224138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:33313 (local address 127.0.197.65:57701)
0929 16:42:53.224908 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:53.224955 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:53.226671 (+ 1716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:53.227681 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:53.227691 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:53.228260 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:53.229015 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:53.229036 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:53.230164 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:53.230169 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:53.231583 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:53.231596 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:53.232385 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:53.232392 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:53.232642 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:53.233395 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:53.233420 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:53.236166 (+ 2746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:42:53.239424 (+ 3258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:53.239434 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:53.239456 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:53.239932 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:53.240338 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:53.240343 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:53.240347 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:53.240562 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:53.241104 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:42:53.241113 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:42:53.241508 (+ 395us) client_negotiation.cc:769] Sending connection context
0929 16:42:53.241784 (+ 276us) client_negotiation.cc:240] Negotiation successful
0929 16:42:53.242369 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":333,"threads_started":1}
I20260929 16:42:53.245436 11352 heartbeater.cc:347] Connected to a master server at 127.0.197.126:33313
I20260929 16:42:53.245807 11352 heartbeater.cc:464] Registering TS with master...
I20260929 16:42:53.246600 11352 heartbeater.cc:511] Master 127.0.197.126:33313 requested a full tablet report, sending...
I20260929 16:42:53.249279 11162 ts_manager.cc:194] Registered new tserver with Master: 69508395076f45daa3148415ef9b52d5 (127.0.197.65:39085)
I20260929 16:42:53.251667 11162 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.65@KRBTEST.COM'} at 127.0.197.65:57701
WARNING: no policy specified for kudu/127.0.197.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab.
Entry for principal kudu/127.0.197.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab.
I20260929 16:42:53.288261 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.66:0
--local_ip_for_outbound_sockets=127.0.197.66
--webserver_interface=127.0.197.66
--webserver_port=0
--tserver_master_addrs=127.0.197.126:33313
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab
--principal=kudu/127.0.197.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:38647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:42:53.632849 11358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:42:53.633153 11358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:42:53.633317 11358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:42:53.641455 11358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:42:53.641598 11358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:42:53.641788 11358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.66
I20260929 16:42:53.652745 11358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:38647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab
--principal=kudu/127.0.197.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.197.66
--webserver_port=0
--tserver_master_addrs=127.0.197.126:33313
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.66
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:42:53.654211 11358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:42:53.655687 11358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:42:53.670179 11366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:53.670143 11363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:53.670220 11364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:42:53.671248 11358 server_base.cc:1034] running on GCE node
I20260929 16:42:53.672039 11358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:42:53.673084 11358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:42:53.674363 11358 hybrid_clock.cc:648] HybridClock initialized: now 1790700173674307 us; error 49 us; skew 500 ppm
I20260929 16:42:53.674888 11358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:42:53 dist-test-slave-l31f krb5kdc[11117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:42:53.678344 11358 init.cc:377] Logged in from keytab as kudu/127.0.197.66@KRBTEST.COM (short username kudu)
I20260929 16:42:53.680210 11358 webserver.cc:460] Webserver started at http://127.0.197.66:40991/ using document root <none> and password file <none>
I20260929 16:42:53.680972 11358 fs_manager.cc:359] Metadata directory not provided
I20260929 16:42:53.681085 11358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:42:53.681463 11358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:42:53.684798 11358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/ts-1/data/instance:
uuid: "8f3b66ec36aa455dbba426faf30da8cb"
format_stamp: "Formatted at 2026-09-29 16:42:53 on dist-test-slave-l31f"
server_key: "2fa25bbee330aeb237a359a0c83e4bc2"
server_key_iv: "f6830b73c254ac8fb90c25262b67d85d"
server_key_version: "encryptionkey@0"
I20260929 16:42:53.685608 11358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/ts-1/wal/instance:
uuid: "8f3b66ec36aa455dbba426faf30da8cb"
format_stamp: "Formatted at 2026-09-29 16:42:53 on dist-test-slave-l31f"
server_key: "2fa25bbee330aeb237a359a0c83e4bc2"
server_key_iv: "f6830b73c254ac8fb90c25262b67d85d"
server_key_version: "encryptionkey@0"
I20260929 16:42:53.691833 11358 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260929 16:42:53.696012 11373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:53.697747 11358 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:42:53.697963 11358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/ts-1/wal
uuid: "8f3b66ec36aa455dbba426faf30da8cb"
format_stamp: "Formatted at 2026-09-29 16:42:53 on dist-test-slave-l31f"
server_key: "2fa25bbee330aeb237a359a0c83e4bc2"
server_key_iv: "f6830b73c254ac8fb90c25262b67d85d"
server_key_version: "encryptionkey@0"
I20260929 16:42:53.698207 11358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:42:53.717612 11358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:42:53.722299 11358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:42:53.722666 11358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:42:53.723749 11358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:42:53.725314 11358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:42:53.725396 11358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:53.725498 11358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:42:53.725560 11358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:53.764441 11358 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.66:44775
I20260929 16:42:53.764504 11485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.66:44775 every 8 connection(s)
I20260929 16:42:53.765813 11358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
I20260929 16:42:53.773939 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 11358
I20260929 16:42:53.774214 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/ts-1/wal/instance
I20260929 16:42:53.774636 789 external_mini_cluster.cc:1428] Setting key 05887194c91a84981d89738ae21461e8
Sep 29 16:42:53 dist-test-slave-l31f krb5kdc[11117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.66@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:42:53.787261 11488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:53.768541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:43781 (local address 127.0.197.126:33313)
0929 16:42:53.769061 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:53.769066 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:53.770110 (+ 1044us) server_negotiation.cc:407] Connection header received
0929 16:42:53.772120 (+ 2010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:53.772126 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:53.772225 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:53.772349 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:53.774004 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:53.774897 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:53.776854 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:53.777284 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:53.777430 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:53.781112 (+ 3682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:53.781118 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:53.781121 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:53.781156 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:53.784064 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:53.784888 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:53.784900 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:53.784904 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:53.785027 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:53.785475 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:53.785486 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:53.785489 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:53.785748 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:53.785906 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:53.786615 (+ 709us) server_negotiation.cc:299] Negotiation successful
0929 16:42:53.786844 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":267,"threads_started":1}
I20260929 16:42:53.788293 11489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:53.769147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:33313 (local address 127.0.197.66:43781)
0929 16:42:53.769918 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:53.769958 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:53.771804 (+ 1846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:53.772644 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:53.772653 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:53.773205 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:53.773809 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:53.773827 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:53.775107 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:53.775111 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:53.776706 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:53.776720 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:53.777532 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:53.777540 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:53.777777 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:53.778499 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:53.778529 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:53.780829 (+ 2300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:42:53.784255 (+ 3426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:53.784273 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:53.784289 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:53.784744 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:53.785143 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:53.785148 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:53.785152 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:53.785347 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:53.785876 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:42:53.785885 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:42:53.786256 (+ 371us) client_negotiation.cc:769] Sending connection context
0929 16:42:53.786534 (+ 278us) client_negotiation.cc:240] Negotiation successful
0929 16:42:53.787143 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":312,"threads_started":1}
I20260929 16:42:53.789861 11486 heartbeater.cc:347] Connected to a master server at 127.0.197.126:33313
I20260929 16:42:53.790213 11486 heartbeater.cc:464] Registering TS with master...
I20260929 16:42:53.791217 11486 heartbeater.cc:511] Master 127.0.197.126:33313 requested a full tablet report, sending...
I20260929 16:42:53.792996 11162 ts_manager.cc:194] Registered new tserver with Master: 8f3b66ec36aa455dbba426faf30da8cb (127.0.197.66:44775)
I20260929 16:42:53.794356 11162 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.66@KRBTEST.COM'} at 127.0.197.66:43781
WARNING: no policy specified for kudu/127.0.197.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab.
Entry for principal kudu/127.0.197.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab.
I20260929 16:42:53.831210 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.67:0
--local_ip_for_outbound_sockets=127.0.197.67
--webserver_interface=127.0.197.67
--webserver_port=0
--tserver_master_addrs=127.0.197.126:33313
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab
--principal=kudu/127.0.197.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:38647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:42:54.173786 11492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:42:54.174079 11492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:42:54.174252 11492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:42:54.182475 11492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:42:54.182679 11492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:42:54.182855 11492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.67
I20260929 16:42:54.193017 11492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:38647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab
--principal=kudu/127.0.197.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.197.67
--webserver_port=0
--tserver_master_addrs=127.0.197.126:33313
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.67
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:42:54.194458 11492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:42:54.195952 11492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:42:54.210417 11500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:54.210593 11497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:42:54.210528 11498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:42:54.211418 11492 server_base.cc:1034] running on GCE node
I20260929 16:42:54.212131 11492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:42:54.213296 11492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:42:54.214545 11492 hybrid_clock.cc:648] HybridClock initialized: now 1790700174214502 us; error 37 us; skew 500 ppm
I20260929 16:42:54.214860 11492 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:42:54 dist-test-slave-l31f krb5kdc[11117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:42:54.218358 11492 init.cc:377] Logged in from keytab as kudu/127.0.197.67@KRBTEST.COM (short username kudu)
I20260929 16:42:54.224385 11492 webserver.cc:460] Webserver started at http://127.0.197.67:33449/ using document root <none> and password file <none>
I20260929 16:42:54.225183 11492 fs_manager.cc:359] Metadata directory not provided
I20260929 16:42:54.225302 11492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:42:54.225658 11492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:42:54.228855 11492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/ts-2/data/instance:
uuid: "d247329769de4b25b035a8e3b35a9193"
format_stamp: "Formatted at 2026-09-29 16:42:54 on dist-test-slave-l31f"
server_key: "dd69c0265e1e1dfa508602841e7c00f7"
server_key_iv: "1260e548cfa2af567c6350eeed5f8d6c"
server_key_version: "encryptionkey@0"
I20260929 16:42:54.229935 11492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/ts-2/wal/instance:
uuid: "d247329769de4b25b035a8e3b35a9193"
format_stamp: "Formatted at 2026-09-29 16:42:54 on dist-test-slave-l31f"
server_key: "dd69c0265e1e1dfa508602841e7c00f7"
server_key_iv: "1260e548cfa2af567c6350eeed5f8d6c"
server_key_version: "encryptionkey@0"
I20260929 16:42:54.236073 11492 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260929 16:42:54.240144 11507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:54.241874 11492 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 16:42:54.242092 11492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/ts-2/wal
uuid: "d247329769de4b25b035a8e3b35a9193"
format_stamp: "Formatted at 2026-09-29 16:42:54 on dist-test-slave-l31f"
server_key: "dd69c0265e1e1dfa508602841e7c00f7"
server_key_iv: "1260e548cfa2af567c6350eeed5f8d6c"
server_key_version: "encryptionkey@0"
I20260929 16:42:54.242311 11492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:42:54.255375 11352 heartbeater.cc:503] Master 127.0.197.126:33313 was elected leader, sending a full tablet report...
I20260929 16:42:54.276839 11492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:42:54.281544 11492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:42:54.281867 11492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:42:54.283289 11492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:42:54.284819 11492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:42:54.284902 11492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:54.285044 11492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:42:54.285099 11492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:42:54.320297 11492 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.67:38625
I20260929 16:42:54.320343 11619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.67:38625 every 8 connection(s)
I20260929 16:42:54.321672 11492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
I20260929 16:42:54.326898 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 11492
I20260929 16:42:54.327176 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/ts-2/wal/instance
I20260929 16:42:54.327916 789 external_mini_cluster.cc:1428] Setting key f743ea0c743437d07aac28ae34562add
Sep 29 16:42:54 dist-test-slave-l31f krb5kdc[11117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.67@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:42:54.343509 11622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:54.324541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:47407 (local address 127.0.197.126:33313)
0929 16:42:54.325045 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:54.325050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:54.325837 (+ 787us) server_negotiation.cc:407] Connection header received
0929 16:42:54.328304 (+ 2467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:54.328310 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:54.328413 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:54.328559 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:54.330560 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:54.331484 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:54.333114 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:54.333537 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:54.333676 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:54.337365 (+ 3689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:54.337370 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:54.337374 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:54.337416 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:54.340067 (+ 2651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:54.340994 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:54.341002 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:54.341005 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:54.341136 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:54.341601 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:54.341606 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:54.341609 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:54.341896 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:54.342066 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:54.342811 (+ 745us) server_negotiation.cc:299] Negotiation successful
0929 16:42:54.343073 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":263,"threads_started":1}
I20260929 16:42:54.344660 11623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:54.324971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:33313 (local address 127.0.197.67:47407)
0929 16:42:54.325667 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:54.325708 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:54.327585 (+ 1877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:54.328848 (+ 1263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:54.328857 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:54.329448 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:42:54.330341 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:54.330368 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:54.331652 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:54.331657 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:54.332948 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:54.332961 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:54.333799 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:54.333806 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:54.334055 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:54.334739 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:42:54.334767 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:42:54.337095 (+ 2328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:42:54.340251 (+ 3156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:54.340262 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:54.340283 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:54.340826 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:54.341260 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:42:54.341265 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:42:54.341269 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:42:54.341480 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:42:54.342034 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:42:54.342043 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:42:54.342426 (+ 383us) client_negotiation.cc:769] Sending connection context
0929 16:42:54.342716 (+ 290us) client_negotiation.cc:240] Negotiation successful
0929 16:42:54.343336 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":279,"threads_started":1}
I20260929 16:42:54.346302 11620 heartbeater.cc:347] Connected to a master server at 127.0.197.126:33313
I20260929 16:42:54.346658 11620 heartbeater.cc:464] Registering TS with master...
I20260929 16:42:54.347532 11620 heartbeater.cc:511] Master 127.0.197.126:33313 requested a full tablet report, sending...
I20260929 16:42:54.349165 11162 ts_manager.cc:194] Registered new tserver with Master: d247329769de4b25b035a8e3b35a9193 (127.0.197.67:38625)
I20260929 16:42:54.350031 11162 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.67@KRBTEST.COM'} at 127.0.197.67:47407
I20260929 16:42:54.356508 789 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:42:54.373955 11622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:54.361139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57276 (local address 127.0.197.126:33313)
0929 16:42:54.361350 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:54.361359 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:54.361767 (+ 408us) server_negotiation.cc:407] Connection header received
0929 16:42:54.361983 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:54.361987 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:54.362052 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:54.362162 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:54.363680 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:54.364442 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:54.365727 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:54.366126 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:54.366259 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:54.368691 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:54.368696 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:54.368699 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:54.368735 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:54.371225 (+ 2490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:54.371897 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:54.371902 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:54.371908 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:54.372034 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:54.372412 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:54.372417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:54.372420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:54.372763 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:54.373022 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:54.373377 (+ 355us) server_negotiation.cc:299] Negotiation successful
0929 16:42:54.373626 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 16:42:54.378213 11162 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57276:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:42:54.385300 11162 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:42:54.410625 11633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:54.399182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.66:44775 (local address 127.0.0.1:39410)
0929 16:42:54.400448 (+ 1266us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:54.400532 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:54.401094 (+ 562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:54.402043 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:54.402046 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:54.402062 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:42:54.402587 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:54.402604 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:54.404916 (+ 2312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:54.404921 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:54.407276 (+ 2355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:54.407293 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:54.409543 (+ 2250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:54.409549 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:54.409770 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:54.409832 (+ 62us) client_negotiation.cc:769] Sending connection context
0929 16:42:54.409984 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 16:42:54.410190 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":981,"thread_start_us":1300,"threads_started":1}
I20260929 16:42:54.410725 11635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:54.399341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39410 (local address 127.0.197.66:44775)
0929 16:42:54.399974 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:54.399984 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:54.400625 (+ 641us) server_negotiation.cc:407] Connection header received
0929 16:42:54.401288 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:54.401296 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:54.401540 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:54.401717 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:42:54.403041 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:54.404744 (+ 1703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:54.407473 (+ 2729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:54.409055 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:54.409186 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:54.409408 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:54.410156 (+ 748us) server_negotiation.cc:299] Negotiation successful
0929 16:42:54.410344 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":338,"threads_started":1}
I20260929 16:42:54.411628 11636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:54.399800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43236 (local address 127.0.197.65:39085)
0929 16:42:54.401221 (+ 1421us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:54.401229 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:54.401254 (+ 25us) server_negotiation.cc:407] Connection header received
0929 16:42:54.401331 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:54.401337 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:54.401540 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:54.401717 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:42:54.403982 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:54.405600 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:54.408638 (+ 3038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:54.409753 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:54.409850 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:54.410208 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:54.411099 (+ 891us) server_negotiation.cc:299] Negotiation successful
0929 16:42:54.411281 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1244,"thread_start_us":1077,"threads_started":1}
I20260929 16:42:54.411665 11634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:54.399259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:39085 (local address 127.0.0.1:43236)
0929 16:42:54.400448 (+ 1189us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:54.400680 (+ 232us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:54.400853 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:54.402010 (+ 1157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:54.402019 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:54.402050 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:42:54.403723 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:54.403735 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:54.405747 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:54.405753 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:54.408420 (+ 2667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:54.408433 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:54.410553 (+ 2120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:54.410560 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:54.410783 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:54.410817 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 16:42:54.410996 (+ 179us) client_negotiation.cc:240] Negotiation successful
0929 16:42:54.411159 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1020,"mutex_wait_us":257,"spinlock_wait_cycles":1283200,"thread_start_us":904,"threads_started":1}
I20260929 16:42:54.412307 11638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:54.401593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54248 (local address 127.0.197.67:38625)
0929 16:42:54.402992 (+ 1399us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:54.403041 (+ 49us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:54.403078 (+ 37us) server_negotiation.cc:407] Connection header received
0929 16:42:54.403621 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:54.403631 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:54.403897 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:54.404105 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:42:54.405024 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:54.406288 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:54.408849 (+ 2561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:54.410289 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:54.410422 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:54.411618 (+ 1196us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:54.411704 (+ 86us) server_negotiation.cc:299] Negotiation successful
0929 16:42:54.411863 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1231,"thread_start_us":1074,"threads_started":1}
I20260929 16:42:54.412884 11637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:54.401023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.67:38625 (local address 127.0.0.1:54248)
0929 16:42:54.402605 (+ 1582us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:54.402626 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:54.403390 (+ 764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:54.404437 (+ 1047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:54.404443 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:54.404468 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:42:54.404814 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:54.404825 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:54.406443 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:54.406451 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:54.408642 (+ 2191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:54.408656 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:54.410735 (+ 2079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:54.410741 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:54.411070 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:54.411104 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 16:42:54.411189 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 16:42:54.412514 (+ 1325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1440,"mutex_wait_us":54,"thread_start_us":1276,"threads_started":1}
I20260929 16:42:54.415337 11421 tablet_service.cc:1467] Processing CreateTablet for tablet 8b18b1591a2b4acebc7b5c8f9fe12c3e (DEFAULT_TABLE table=test-table [id=cc29765bfdcc4ba0b913511f5493f1f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:42:54.415372 11287 tablet_service.cc:1467] Processing CreateTablet for tablet 8b18b1591a2b4acebc7b5c8f9fe12c3e (DEFAULT_TABLE table=test-table [id=cc29765bfdcc4ba0b913511f5493f1f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:42:54.416628 11287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b18b1591a2b4acebc7b5c8f9fe12c3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:42:54.416632 11421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b18b1591a2b4acebc7b5c8f9fe12c3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:42:54.422163 11555 tablet_service.cc:1467] Processing CreateTablet for tablet 8b18b1591a2b4acebc7b5c8f9fe12c3e (DEFAULT_TABLE table=test-table [id=cc29765bfdcc4ba0b913511f5493f1f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:42:54.423846 11555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b18b1591a2b4acebc7b5c8f9fe12c3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:42:54.428764 11639 tablet_bootstrap.cc:492] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P 8f3b66ec36aa455dbba426faf30da8cb: Bootstrap starting.
I20260929 16:42:54.430960 11640 tablet_bootstrap.cc:492] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P 69508395076f45daa3148415ef9b52d5: Bootstrap starting.
I20260929 16:42:54.432993 11639 tablet_bootstrap.cc:654] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P 8f3b66ec36aa455dbba426faf30da8cb: Neither blocks nor log segments found. Creating new log.
I20260929 16:42:54.434403 11639 log.cc:824] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P 8f3b66ec36aa455dbba426faf30da8cb: Log is configured to *not* fsync() on all Append() calls
I20260929 16:42:54.434952 11640 tablet_bootstrap.cc:654] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P 69508395076f45daa3148415ef9b52d5: Neither blocks nor log segments found. Creating new log.
I20260929 16:42:54.436293 11640 log.cc:824] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P 69508395076f45daa3148415ef9b52d5: Log is configured to *not* fsync() on all Append() calls
I20260929 16:42:54.437903 11639 tablet_bootstrap.cc:492] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P 8f3b66ec36aa455dbba426faf30da8cb: No bootstrap required, opened a new log
I20260929 16:42:54.438212 11639 ts_tablet_manager.cc:1397] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P 8f3b66ec36aa455dbba426faf30da8cb: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.002s
I20260929 16:42:54.439226 11643 tablet_bootstrap.cc:492] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P d247329769de4b25b035a8e3b35a9193: Bootstrap starting.
I20260929 16:42:54.440110 11640 tablet_bootstrap.cc:492] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P 69508395076f45daa3148415ef9b52d5: No bootstrap required, opened a new log
I20260929 16:42:54.440399 11640 ts_tablet_manager.cc:1397] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P 69508395076f45daa3148415ef9b52d5: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260929 16:42:54.442724 11643 tablet_bootstrap.cc:654] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P d247329769de4b25b035a8e3b35a9193: Neither blocks nor log segments found. Creating new log.
I20260929 16:42:54.444036 11643 log.cc:824] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P d247329769de4b25b035a8e3b35a9193: Log is configured to *not* fsync() on all Append() calls
I20260929 16:42:54.447423 11643 tablet_bootstrap.cc:492] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P d247329769de4b25b035a8e3b35a9193: No bootstrap required, opened a new log
I20260929 16:42:54.447741 11643 ts_tablet_manager.cc:1397] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P d247329769de4b25b035a8e3b35a9193: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260929 16:42:54.449126 11639 raft_consensus.cc:348] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P 8f3b66ec36aa455dbba426faf30da8cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3b66ec36aa455dbba426faf30da8cb" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44775 } } peers { permanent_uuid: "69508395076f45daa3148415ef9b52d5" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 39085 } } peers { permanent_uuid: "d247329769de4b25b035a8e3b35a9193" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 38625 } }
I20260929 16:42:54.449656 11639 raft_consensus.cc:374] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P 8f3b66ec36aa455dbba426faf30da8cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:42:54.449796 11639 raft_consensus.cc:729] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P 8f3b66ec36aa455dbba426faf30da8cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f3b66ec36aa455dbba426faf30da8cb, State: Initialized, Role: FOLLOWER
I20260929 16:42:54.450520 11639 consensus_queue.cc:260] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P 8f3b66ec36aa455dbba426faf30da8cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3b66ec36aa455dbba426faf30da8cb" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44775 } } peers { permanent_uuid: "69508395076f45daa3148415ef9b52d5" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 39085 } } peers { permanent_uuid: "d247329769de4b25b035a8e3b35a9193" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 38625 } }
I20260929 16:42:54.451273 11640 raft_consensus.cc:348] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P 69508395076f45daa3148415ef9b52d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3b66ec36aa455dbba426faf30da8cb" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44775 } } peers { permanent_uuid: "69508395076f45daa3148415ef9b52d5" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 39085 } } peers { permanent_uuid: "d247329769de4b25b035a8e3b35a9193" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 38625 } }
I20260929 16:42:54.451794 11640 raft_consensus.cc:374] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P 69508395076f45daa3148415ef9b52d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:42:54.451782 11486 heartbeater.cc:503] Master 127.0.197.126:33313 was elected leader, sending a full tablet report...
I20260929 16:42:54.451956 11640 raft_consensus.cc:729] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P 69508395076f45daa3148415ef9b52d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69508395076f45daa3148415ef9b52d5, State: Initialized, Role: FOLLOWER
I20260929 16:42:54.452616 11639 ts_tablet_manager.cc:1428] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P 8f3b66ec36aa455dbba426faf30da8cb: Time spent starting tablet: real 0.014s user 0.012s sys 0.004s
I20260929 16:42:54.452874 11640 consensus_queue.cc:260] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P 69508395076f45daa3148415ef9b52d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3b66ec36aa455dbba426faf30da8cb" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44775 } } peers { permanent_uuid: "69508395076f45daa3148415ef9b52d5" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 39085 } } peers { permanent_uuid: "d247329769de4b25b035a8e3b35a9193" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 38625 } }
I20260929 16:42:54.454806 11640 ts_tablet_manager.cc:1428] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P 69508395076f45daa3148415ef9b52d5: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260929 16:42:54.457792 11643 raft_consensus.cc:348] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P d247329769de4b25b035a8e3b35a9193 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3b66ec36aa455dbba426faf30da8cb" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44775 } } peers { permanent_uuid: "69508395076f45daa3148415ef9b52d5" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 39085 } } peers { permanent_uuid: "d247329769de4b25b035a8e3b35a9193" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 38625 } }
I20260929 16:42:54.458205 11643 raft_consensus.cc:374] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P d247329769de4b25b035a8e3b35a9193 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:42:54.458335 11643 raft_consensus.cc:729] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P d247329769de4b25b035a8e3b35a9193 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d247329769de4b25b035a8e3b35a9193, State: Initialized, Role: FOLLOWER
I20260929 16:42:54.459045 11643 consensus_queue.cc:260] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P d247329769de4b25b035a8e3b35a9193 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3b66ec36aa455dbba426faf30da8cb" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44775 } } peers { permanent_uuid: "69508395076f45daa3148415ef9b52d5" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 39085 } } peers { permanent_uuid: "d247329769de4b25b035a8e3b35a9193" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 38625 } }
I20260929 16:42:54.460260 11620 heartbeater.cc:503] Master 127.0.197.126:33313 was elected leader, sending a full tablet report...
I20260929 16:42:54.460973 11643 ts_tablet_manager.cc:1428] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P d247329769de4b25b035a8e3b35a9193: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
W20260929 16:42:54.473449 11353 tablet.cc:2369] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P 69508395076f45daa3148415ef9b52d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:42:54.517970 11487 tablet.cc:2369] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P 8f3b66ec36aa455dbba426faf30da8cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:42:54.573357 11621 tablet.cc:2369] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P d247329769de4b25b035a8e3b35a9193: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:42:54.660482 11646 raft_consensus.cc:482] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P 69508395076f45daa3148415ef9b52d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:42:54.660840 11646 raft_consensus.cc:504] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P 69508395076f45daa3148415ef9b52d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3b66ec36aa455dbba426faf30da8cb" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44775 } } peers { permanent_uuid: "69508395076f45daa3148415ef9b52d5" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 39085 } } peers { permanent_uuid: "d247329769de4b25b035a8e3b35a9193" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 38625 } }
I20260929 16:42:54.662854 11646 leader_election.cc:288] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P 69508395076f45daa3148415ef9b52d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f3b66ec36aa455dbba426faf30da8cb (127.0.197.66:44775), d247329769de4b25b035a8e3b35a9193 (127.0.197.67:38625)
I20260929 16:42:54.670570 11635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:54.663478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:53099 (local address 127.0.197.66:44775)
0929 16:42:54.663681 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:54.663686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:54.664178 (+ 492us) server_negotiation.cc:407] Connection header received
0929 16:42:54.664430 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:54.664436 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:54.664540 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:54.664657 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:42:54.665342 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:54.666330 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:54.668482 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:54.669617 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:54.669749 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:54.669806 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:54.670258 (+ 452us) server_negotiation.cc:299] Negotiation successful
0929 16:42:54.670363 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 16:42:54.670881 11648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:54.663280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.66:44775 (local address 127.0.197.65:53099)
0929 16:42:54.664079 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:54.664098 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:54.664247 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:54.664706 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:54.664714 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:54.664744 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:42:54.665154 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:54.665169 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:54.666477 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:54.666483 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:54.668302 (+ 1819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:54.668316 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:54.669855 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:54.669861 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:54.670045 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:54.670082 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 16:42:54.670201 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 16:42:54.670354 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":434,"threads_started":1}
I20260929 16:42:54.671871 11649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:54.664177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.67:38625 (local address 127.0.197.65:51111)
0929 16:42:54.664867 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:54.664891 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 16:42:54.664991 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:42:54.665344 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:42:54.665348 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:42:54.665373 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:42:54.665752 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:54.665766 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:54.667111 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:54.667115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:54.668943 (+ 1828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:42:54.668958 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:42:54.670956 (+ 1998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:42:54.670964 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:42:54.671226 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:54.671265 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 16:42:54.671379 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 16:42:54.671529 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":396,"threads_started":1}
I20260929 16:42:54.671917 11638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:54.664349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:51111 (local address 127.0.197.67:38625)
0929 16:42:54.664574 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:54.664581 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:54.664984 (+ 403us) server_negotiation.cc:407] Connection header received
0929 16:42:54.665093 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:54.665100 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:54.665180 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:54.665275 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:42:54.665937 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:54.666863 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:54.669376 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:54.670506 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:54.670673 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:54.670748 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:54.671413 (+ 665us) server_negotiation.cc:299] Negotiation successful
0929 16:42:54.671665 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260929 16:42:54.672235 11441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b18b1591a2b4acebc7b5c8f9fe12c3e" candidate_uuid: "69508395076f45daa3148415ef9b52d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f3b66ec36aa455dbba426faf30da8cb" is_pre_election: true
I20260929 16:42:54.672845 11441 raft_consensus.cc:2454] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P 8f3b66ec36aa455dbba426faf30da8cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69508395076f45daa3148415ef9b52d5 in term 0.
I20260929 16:42:54.673058 11575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b18b1591a2b4acebc7b5c8f9fe12c3e" candidate_uuid: "69508395076f45daa3148415ef9b52d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d247329769de4b25b035a8e3b35a9193" is_pre_election: true
I20260929 16:42:54.673542 11575 raft_consensus.cc:2454] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P d247329769de4b25b035a8e3b35a9193 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69508395076f45daa3148415ef9b52d5 in term 0.
I20260929 16:42:54.673590 11242 leader_election.cc:302] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P 69508395076f45daa3148415ef9b52d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69508395076f45daa3148415ef9b52d5, 8f3b66ec36aa455dbba426faf30da8cb; no voters:
I20260929 16:42:54.674034 11646 raft_consensus.cc:2781] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P 69508395076f45daa3148415ef9b52d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:42:54.674198 11646 raft_consensus.cc:482] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P 69508395076f45daa3148415ef9b52d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:42:54.674307 11646 raft_consensus.cc:3037] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P 69508395076f45daa3148415ef9b52d5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:42:54.676630 11646 raft_consensus.cc:504] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P 69508395076f45daa3148415ef9b52d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3b66ec36aa455dbba426faf30da8cb" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44775 } } peers { permanent_uuid: "69508395076f45daa3148415ef9b52d5" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 39085 } } peers { permanent_uuid: "d247329769de4b25b035a8e3b35a9193" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 38625 } }
I20260929 16:42:54.677299 11646 leader_election.cc:288] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P 69508395076f45daa3148415ef9b52d5 [CANDIDATE]: Term 1 election: Requested vote from peers 8f3b66ec36aa455dbba426faf30da8cb (127.0.197.66:44775), d247329769de4b25b035a8e3b35a9193 (127.0.197.67:38625)
I20260929 16:42:54.677847 11575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b18b1591a2b4acebc7b5c8f9fe12c3e" candidate_uuid: "69508395076f45daa3148415ef9b52d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d247329769de4b25b035a8e3b35a9193"
I20260929 16:42:54.677872 11441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b18b1591a2b4acebc7b5c8f9fe12c3e" candidate_uuid: "69508395076f45daa3148415ef9b52d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f3b66ec36aa455dbba426faf30da8cb"
I20260929 16:42:54.678082 11575 raft_consensus.cc:3037] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P d247329769de4b25b035a8e3b35a9193 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:42:54.678090 11441 raft_consensus.cc:3037] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P 8f3b66ec36aa455dbba426faf30da8cb [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:42:54.680519 11441 raft_consensus.cc:2454] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P 8f3b66ec36aa455dbba426faf30da8cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69508395076f45daa3148415ef9b52d5 in term 1.
I20260929 16:42:54.680619 11575 raft_consensus.cc:2454] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P d247329769de4b25b035a8e3b35a9193 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69508395076f45daa3148415ef9b52d5 in term 1.
I20260929 16:42:54.681154 11242 leader_election.cc:302] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P 69508395076f45daa3148415ef9b52d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69508395076f45daa3148415ef9b52d5, d247329769de4b25b035a8e3b35a9193; no voters:
I20260929 16:42:54.681470 11646 raft_consensus.cc:2781] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P 69508395076f45daa3148415ef9b52d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:42:54.681993 11646 raft_consensus.cc:686] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P 69508395076f45daa3148415ef9b52d5 [term 1 LEADER]: Becoming Leader. State: Replica: 69508395076f45daa3148415ef9b52d5, State: Running, Role: LEADER
I20260929 16:42:54.682492 11646 consensus_queue.cc:237] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P 69508395076f45daa3148415ef9b52d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3b66ec36aa455dbba426faf30da8cb" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44775 } } peers { permanent_uuid: "69508395076f45daa3148415ef9b52d5" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 39085 } } peers { permanent_uuid: "d247329769de4b25b035a8e3b35a9193" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 38625 } }
I20260929 16:42:54.687554 11162 catalog_manager.cc:5495] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P 69508395076f45daa3148415ef9b52d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69508395076f45daa3148415ef9b52d5 (127.0.197.65). New cstate: current_term: 1 leader_uuid: "69508395076f45daa3148415ef9b52d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3b66ec36aa455dbba426faf30da8cb" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69508395076f45daa3148415ef9b52d5" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 39085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d247329769de4b25b035a8e3b35a9193" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 38625 } health_report { overall_health: UNKNOWN } } }
I20260929 16:42:54.712107 11636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:54.706633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43244 (local address 127.0.197.65:39085)
0929 16:42:54.706861 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:54.706866 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:54.706886 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:42:54.707130 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:54.707134 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:54.707229 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:54.707398 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:42:54.707765 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:54.708521 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:54.709814 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:54.710242 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:54.710380 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:54.710923 (+ 543us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:42:54.711484 (+ 561us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:42:54.711645 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:54.711749 (+ 104us) server_negotiation.cc:299] Negotiation successful
0929 16:42:54.711832 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 16:42:54.720716 11575 raft_consensus.cc:1261] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P d247329769de4b25b035a8e3b35a9193 [term 1 FOLLOWER]: Refusing update from remote peer 69508395076f45daa3148415ef9b52d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:42:54.720716 11441 raft_consensus.cc:1261] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P 8f3b66ec36aa455dbba426faf30da8cb [term 1 FOLLOWER]: Refusing update from remote peer 69508395076f45daa3148415ef9b52d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:42:54.721777 11650 consensus_queue.cc:1035] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P 69508395076f45daa3148415ef9b52d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f3b66ec36aa455dbba426faf30da8cb" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:42:54.722178 11646 consensus_queue.cc:1035] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P 69508395076f45daa3148415ef9b52d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d247329769de4b25b035a8e3b35a9193" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 38625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:42:54.736222 11654 mvcc.cc:205] Tried to move back new op lower bound from 7334707915645743104 to 7334707915503960064. Current Snapshot: MvccSnapshot[applied={T|T < 7334707915645743104}]
I20260929 16:42:54.752978 11161 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:42:54.753306 11161 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:42:54.757104 11161 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ee2768f3c2b42bc8dbfc53d4ee2bed6: Sending DeleteTablet for 3 replicas of tablet 8b18b1591a2b4acebc7b5c8f9fe12c3e
I20260929 16:42:54.758526 11287 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b18b1591a2b4acebc7b5c8f9fe12c3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:42:54 UTC) from {username='kudu', principal='kudu/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:43236
I20260929 16:42:54.758526 11421 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b18b1591a2b4acebc7b5c8f9fe12c3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:42:54 UTC) from {username='kudu', principal='kudu/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:39410
I20260929 16:42:54.758916 11555 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b18b1591a2b4acebc7b5c8f9fe12c3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:42:54 UTC) from {username='kudu', principal='kudu/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:54248
I20260929 16:42:54.760565 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 11224
I20260929 16:42:54.769866 11662 tablet_replica.cc:330] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P 69508395076f45daa3148415ef9b52d5: stopping tablet replica
I20260929 16:42:54.771046 11662 raft_consensus.cc:2229] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P 69508395076f45daa3148415ef9b52d5 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:42:54.772502 11662 raft_consensus.cc:2258] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P 69508395076f45daa3148415ef9b52d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:42:54.776229 11662 ts_tablet_manager.cc:1905] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P 69508395076f45daa3148415ef9b52d5: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 16:42:54 dist-test-slave-l31f krb5kdc[11117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.65@KRBTEST.COM
I20260929 16:42:54.786895 11663 tablet_replica.cc:330] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P 8f3b66ec36aa455dbba426faf30da8cb: stopping tablet replica
I20260929 16:42:54.787436 11663 raft_consensus.cc:2229] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P 8f3b66ec36aa455dbba426faf30da8cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:42:54.788015 11663 raft_consensus.cc:2258] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P 8f3b66ec36aa455dbba426faf30da8cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:42:54.788796 11664 tablet_replica.cc:330] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P d247329769de4b25b035a8e3b35a9193: stopping tablet replica
I20260929 16:42:54.789242 11664 raft_consensus.cc:2229] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P d247329769de4b25b035a8e3b35a9193 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:42:54.789891 11664 raft_consensus.cc:2258] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P d247329769de4b25b035a8e3b35a9193 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:42:54.790027 11636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:54.761928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43250 (local address 127.0.197.65:39085)
0929 16:42:54.762138 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:54.762144 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:54.767306 (+ 5162us) server_negotiation.cc:407] Connection header received
0929 16:42:54.767520 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:54.767526 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:54.767642 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:54.767756 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:54.771384 (+ 3628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:54.772639 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:54.774623 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:54.775542 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:54.775661 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:54.781131 (+ 5470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:54.781148 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:54.781169 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:54.781225 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:54.785927 (+ 4702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:54.787078 (+ 1151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:54.787089 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:54.787093 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:54.787309 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:54.787870 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:54.787879 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:54.787883 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:54.788348 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:54.788655 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:54.789299 (+ 644us) server_negotiation.cc:299] Negotiation successful
0929 16:42:54.789587 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 16:42:54.791604 11347 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:42:54.794610 11664 ts_tablet_manager.cc:1905] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P d247329769de4b25b035a8e3b35a9193: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:42:54.795125 11663 ts_tablet_manager.cc:1905] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P 8f3b66ec36aa455dbba426faf30da8cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:42:54.801990 11664 ts_tablet_manager.cc:1918] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P d247329769de4b25b035a8e3b35a9193: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:42:54.802325 11664 log.cc:1196] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P d247329769de4b25b035a8e3b35a9193: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/ts-2/wal/wals/8b18b1591a2b4acebc7b5c8f9fe12c3e
I20260929 16:42:54.803205 11664 ts_tablet_manager.cc:1939] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P d247329769de4b25b035a8e3b35a9193: Deleting consensus metadata
I20260929 16:42:54.805516 11663 ts_tablet_manager.cc:1918] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P 8f3b66ec36aa455dbba426faf30da8cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:42:54.805913 11663 log.cc:1196] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P 8f3b66ec36aa455dbba426faf30da8cb: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/ts-1/wal/wals/8b18b1591a2b4acebc7b5c8f9fe12c3e
I20260929 16:42:54.806492 11663 ts_tablet_manager.cc:1939] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P 8f3b66ec36aa455dbba426faf30da8cb: Deleting consensus metadata
I20260929 16:42:54.806727 11148 catalog_manager.cc:4841] TS d247329769de4b25b035a8e3b35a9193 (127.0.197.67:38625): tablet 8b18b1591a2b4acebc7b5c8f9fe12c3e (table test-table [id=cc29765bfdcc4ba0b913511f5493f1f3]) successfully deleted
I20260929 16:42:54.808362 11148 catalog_manager.cc:4841] TS 8f3b66ec36aa455dbba426faf30da8cb (127.0.197.66:44775): tablet 8b18b1591a2b4acebc7b5c8f9fe12c3e (table test-table [id=cc29765bfdcc4ba0b913511f5493f1f3]) successfully deleted
I20260929 16:42:54.904009 11662 ts_tablet_manager.cc:1918] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P 69508395076f45daa3148415ef9b52d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:42:54.904313 11662 log.cc:1196] T 8b18b1591a2b4acebc7b5c8f9fe12c3e P 69508395076f45daa3148415ef9b52d5: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790700068255611-789-0/minicluster-data/ts-0/wal/wals/8b18b1591a2b4acebc7b5c8f9fe12c3e
I20260929 16:42:54.904801 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 11224
I20260929 16:42:54.924367 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 11358
W20260929 16:42:54.924384 11149 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.197.65:39085 (error 108)
I20260929 16:42:54.926355 11668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:54.925639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:39085 (local address 127.0.0.1:43258)
0929 16:42:54.926106 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:54.926204 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.65:39085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":246,"threads_started":1}
W20260929 16:42:54.926580 11149 catalog_manager.cc:4584] TS 69508395076f45daa3148415ef9b52d5 (127.0.197.65:39085): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8b18b1591a2b4acebc7b5c8f9fe12c3e: Network error: Client connection negotiation failed: client connection to 127.0.197.65:39085: connect: Connection refused (error 111)
Sep 29 16:42:54 dist-test-slave-l31f krb5kdc[11117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.66@KRBTEST.COM
I20260929 16:42:54.940445 11635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:54.925221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39418 (local address 127.0.197.66:44775)
0929 16:42:54.925403 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:54.925409 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:54.925431 (+ 22us) server_negotiation.cc:407] Connection header received
0929 16:42:54.925640 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:54.925645 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:54.925718 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:54.925850 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:54.927299 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:54.928017 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:54.929268 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:54.929701 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:54.929857 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:54.934314 (+ 4457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:54.934331 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:54.934347 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:54.934387 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:54.937290 (+ 2903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:54.938100 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:54.938108 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:54.938111 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:54.938241 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:54.938671 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:54.938675 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:54.938677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:54.939200 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:54.939534 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:54.939864 (+ 330us) server_negotiation.cc:299] Negotiation successful
0929 16:42:54.940105 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 16:42:54.941152 11481 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:42:54.980232 11668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:54.979725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:39085 (local address 127.0.0.1:43274)
0929 16:42:54.979982 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:54.980088 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.65:39085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260929 16:42:54.981307 11668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:54.980964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:39085 (local address 127.0.0.1:43286)
0929 16:42:54.981100 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:54.981168 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.65:39085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 16:42:55.039938 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 11358
I20260929 16:42:55.042003 11668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:55.041569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:39085 (local address 127.0.0.1:43302)
0929 16:42:55.041771 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:55.041864 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.65:39085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 16:42:55.043730 11668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:55.042833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:39085 (local address 127.0.0.1:43310)
0929 16:42:55.043090 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:55.043180 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.65:39085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260929 16:42:55.060294 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 11492
Sep 29 16:42:55 dist-test-slave-l31f krb5kdc[11117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.67@KRBTEST.COM
I20260929 16:42:55.077749 11638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:55.061120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54250 (local address 127.0.197.67:38625)
0929 16:42:55.061298 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 16:42:55.061302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:42:55.061321 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:42:55.061376 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:42:55.061380 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:42:55.061450 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:42:55.061544 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:42:55.062899 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:55.063761 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:55.065163 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:42:55.065559 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:42:55.065663 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:42:55.070558 (+ 4895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:42:55.070575 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:42:55.070593 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:42:55.070635 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:42:55.074144 (+ 3509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:55.075142 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:55.075151 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:55.075154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:55.075314 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:42:55.075727 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:42:55.075731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:42:55.075733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:42:55.076124 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:42:55.076343 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 16:42:55.077127 (+ 784us) server_negotiation.cc:299] Negotiation successful
0929 16:42:55.077418 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 16:42:55.078609 11615 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:42:55.124747 11668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:55.124176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:39085 (local address 127.0.0.1:43316)
0929 16:42:55.124450 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:55.124564 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.65:39085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260929 16:42:55.126010 11668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:55.125521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:39085 (local address 127.0.0.1:43320)
0929 16:42:55.125744 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:55.125832 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.65:39085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260929 16:42:55.171650 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 11492
I20260929 16:42:55.189862 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 11130
I20260929 16:42:55.191268 11192 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:42:55.305950 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 11130
I20260929 16:42:55.306005 11668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:42:55.305498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:39085 (local address 127.0.0.1:43336)
0929 16:42:55.305680 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 16:42:55.305822 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.65:39085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
2026-09-29T16:42:55Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4621 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:42:55 dist-test-slave-l31f krb5kdc[11674](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:42:55 dist-test-slave-l31f krb5kdc[11674](info): set up 2 sockets
Sep 29 16:42:55 dist-test-slave-l31f krb5kdc[11674](info): commencing operation
krb5kdc: starting...
W20260929 16:42:56.468166 789 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.006s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/krb5kdc/test-user.keytab.
Sep 29 16:42:56 dist-test-slave-l31f krb5kdc[11674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:42:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:42:56Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.197.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.197.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.197.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/krb5kdc/rangeradmin_127.0.197.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.197.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/krb5kdc/rangeradmin_127.0.197.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.197.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.197.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.197.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/krb5kdc/rangerlookup_127.0.197.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.197.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/krb5kdc/rangerlookup_127.0.197.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.197.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.197.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.197.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/krb5kdc/HTTP_127.0.197.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.197.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/krb5kdc/HTTP_127.0.197.84@KRBTEST.COM.keytab.
I20260929 16:42:56.836257 789 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskQmsv3Z/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/postgres -l logfile start
2026-09-29 16:42:59.824 UTC [11704] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 16:42:59.824 UTC [11704] LOG: listening on IPv4 address "127.0.197.84", port 50321
2026-09-29 16:42:59.829 UTC [11704] LOG: listening on Unix socket "/tmp/.s.PGSQL.50321"
2026-09-29 16:42:59.855 UTC [11707] LOG: database system was shut down at 2026-09-29 16:42:57 UTC
2026-09-29 16:42:59.859 UTC [11704] LOG: database system is ready to accept connections
I20260929 16:43:00.859814 789 mini_postgres.cc:96] Postgres bound to 50321
2026-09-29 16:43:00.878 UTC [11715] FATAL: database "slave" does not exist
127.0.197.84:50321 - accepting connections
I20260929 16:43:00.880053 789 mini_ranger.cc:161] Starting Ranger...
I20260929 16:43:00.912508 789 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 16:43:01.572134 789 mini_ranger.cc:87] Created ranger Postgres database
I20260929 16:43:01.572283 789 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ranger-admin
2026-09-29 16:43:01,900 [I] DB FLAVOR :POSTGRES
2026-09-29 16:43:01,900 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 16:43:01,901 [I] Checking connection..
2026-09-29 16:43:01,901 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQmsv3Z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQmsv3Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.197.84:50321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 16:43:02,173 [I] Checking connection passed.
2026-09-29 16:43:02,173 [I] --------- Verifying version history table ---------
2026-09-29 16:43:02,173 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQmsv3Z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQmsv3Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.197.84:50321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 16:43:02,457 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 16:43:02,457 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQmsv3Z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQmsv3Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.197.84:50321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 16:43:02,709 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 16:43:02,709 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 16:43:02,709 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQmsv3Z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQmsv3Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.197.84:50321/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQmsv3Z/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 16:43:02.954 UTC [11779] WARNING: there is no transaction in progress
2026-09-29 16:43:02,962 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 16:43:02,962 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQmsv3Z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQmsv3Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.197.84:50321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 16:43:03,243 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 16:43:03,244 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 16:43:03,244 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQmsv3Z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQmsv3Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.197.84:50321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 16:43:03,502 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQmsv3Z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQmsv3Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.197.84:50321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 16:43:03,763 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQmsv3Z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQmsv3Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.197.84:50321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-l31f.c.gcp-upstream.internal','N') ;"
2026-09-29 16:43:04,012 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQmsv3Z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQmsv3Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.197.84:50321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 16:43:04,261 [I] Table x_portal_user does not exist in database ranger
2026-09-29 16:43:04,261 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQmsv3Z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQmsv3Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.197.84:50321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 16:43:04,521 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 16:43:04,522 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQmsv3Z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQmsv3Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.197.84:50321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 16:43:04,806 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQmsv3Z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQmsv3Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.197.84:50321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 16:43:05,086 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 16:43:05,086 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQmsv3Z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQmsv3Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.197.84:50321/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQmsv3Z/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 16:43:06.300 UTC [11887] WARNING: there is no transaction in progress
2026-09-29 16:43:06.323 UTC [11887] WARNING: there is no transaction in progress
2026-09-29 16:43:06.344 UTC [11887] WARNING: there is no transaction in progress
2026-09-29 16:43:06.364 UTC [11887] WARNING: there is no transaction in progress
2026-09-29 16:43:06.385 UTC [11887] WARNING: there is no transaction in progress
2026-09-29 16:43:06.410 UTC [11887] WARNING: there is no transaction in progress
2026-09-29 16:43:06.476 UTC [11887] WARNING: there is no transaction in progress
2026-09-29 16:43:06.485 UTC [11887] WARNING: there is no transaction in progress
2026-09-29 16:43:06.504 UTC [11887] WARNING: there is no transaction in progress
2026-09-29 16:43:06.517 UTC [11887] WARNING: there is no transaction in progress
2026-09-29 16:43:06.532 UTC [11887] WARNING: there is no transaction in progress
2026-09-29 16:43:06.550 UTC [11887] WARNING: there is no transaction in progress
2026-09-29 16:43:06.564 UTC [11887] WARNING: there is no transaction in progress
2026-09-29 16:43:06.575 UTC [11887] WARNING: there is no transaction in progress
2026-09-29 16:43:06.586 UTC [11887] WARNING: there is no transaction in progress
2026-09-29 16:43:06.595 UTC [11887] WARNING: there is no transaction in progress
2026-09-29 16:43:07,488 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 16:43:07,489 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQmsv3Z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQmsv3Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.197.84:50321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 16:43:07,740 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 16:43:07,740 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQmsv3Z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQmsv3Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.197.84:50321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 16:43:08,025 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 16:43:08,025 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQmsv3Z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQmsv3Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.197.84:50321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 16:43:08,287 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 16:43:08,287 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQmsv3Z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQmsv3Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.197.84:50321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 16:43:08,538 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQmsv3Z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQmsv3Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.197.84:50321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 16:43:08,813 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQmsv3Z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQmsv3Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.197.84:50321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-l31f.c.gcp-upstream.internal';"
2026-09-29 16:43:09,069 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 16:43:09,070 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQmsv3Z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQmsv3Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.197.84:50321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 16:43:09,331 [I] DB_PATCHES have already been applied
I20260929 16:43:09.337934 789 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ranger-admin:/tmp/dist-test-taskQmsv3Z/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskQmsv3Z/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskQmsv3Z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQmsv3Z/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 16:43:09.338030 789 mini_ranger.cc:193] Using host: 127.0.197.84
I20260929 16:43:09.354354 789 mini_ranger.cc:227] Ranger admin URL: http://127.0.197.84:49719
Sep 29, 2026 4:43:09 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 4:43:09 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 4:43:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 4:43:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 4:43:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 4:43:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 4:43:10 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 16:43:10.516904 789 mini_ranger.cc:160] Time spent starting Ranger: real 9.637s user 0.001s sys 0.045s
Sep 29, 2026 4:43:31 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 4:43:31 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 29, 2026 4:43:31 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 4:43:31 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 4:43:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 4:43:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 4:43:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 4:43:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 4:43:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 4:43:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 4:43:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 4:43:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 4:43:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 4:43:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 4:43:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 4:43:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 4:43:32 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 4:43:33 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 16:43:33.612552 789 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.197.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.197.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.197.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/krb5kdc/rangerkms_127.0.197.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.197.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/krb5kdc/rangerkms_127.0.197.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.197.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.197.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.197.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/krb5kdc/HTTP_127.0.197.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.197.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/krb5kdc/HTTP_127.0.197.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 16:43:33 dist-test-slave-l31f krb5kdc[11674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700213, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 16:43:33.792796 789 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 16:43:33.815851 789 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 16:43:34.892998 789 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 16:43:34.893172 789 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ranger-kms
I20260929 16:43:34.893308 789 mini_ranger_kms.cc:227] Using postgres at 127.0.197.84:50321
2026-09-29 16:43:35,326 [I] DB FLAVOR :POSTGRES
2026-09-29 16:43:35,326 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 16:43:35,326 [I] Checking connection
2026-09-29 16:43:35,326 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQmsv3Z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.197.84:50321/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 16:43:35,597 [I] connection success
2026-09-29 16:43:35,597 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 16:43:35,597 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 16:43:35,598 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQmsv3Z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.197.84:50321/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 16:43:35,879 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 16:43:35,880 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 16:43:35,880 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 16:43:35,880 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQmsv3Z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.197.84:50321/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 16:43:36,195 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 16:43:36.759202 789 mini_ranger_kms.cc:310] Created kms service
I20260929 16:43:36.966763 789 mini_ranger_kms.cc:326] Created kudu user
I20260929 16:43:37.018189 789 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 16:43:37 dist-test-slave-l31f krb5kdc[11674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700213, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.197.84@KRBTEST.COM
I20260929 16:43:37.497606 789 mini_ranger_kms.cc:384] Added ranger policy
I20260929 16:43:37.497956 789 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ranger-kms:/tmp/dist-test-taskQmsv3Z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQmsv3Z/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskQmsv3Z/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 16:43:37.498015 789 mini_ranger_kms.cc:240] Using host: 127.0.197.84
I20260929 16:43:37.524303 789 mini_ranger_kms.cc:276] Ranger KMS PID: 12068
I20260929 16:43:37.524529 789 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.197.84:56653
Sep 29, 2026 4:43:38 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 4:43:38 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 4:43:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 4:43:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 4:43:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 4:43:39 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 29 16:43:39 dist-test-slave-l31f krb5kdc[11674](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790700219, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.197.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:43:39.975999 789 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.183s user 0.093s sys 0.369s
I20260929 16:43:39.976260 789 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 16:43:39.976452 789 mini_ranger_kms.cc:401] 127.0.197.84:56653/kms/v1/keys
Sep 29 16:43:40 dist-test-slave-l31f krb5kdc[11674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700219, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.197.84@KRBTEST.COM for HTTP/127.0.197.84@KRBTEST.COM
Sep 29 16:43:40 dist-test-slave-l31f krb5kdc[11674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700219, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.197.84@KRBTEST.COM for HTTP/127.0.197.84@KRBTEST.COM
Sep 29 16:43:43 dist-test-slave-l31f krb5kdc[11674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.197.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
I20260929 16:43:43.307595 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:34507
--webserver_interface=127.0.197.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:33399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.126:34507
--ranger_config_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.197.84:56653/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:43:43.679865 12118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:43:43.680202 12118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:43:43.680295 12118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:43:43.688256 12118 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:43:43.688393 12118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:43:43.688439 12118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:43:43.688488 12118 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:43:43.688537 12118 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:43:43.700309 12118 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:33399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.197.84:56653/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.197.126:34507
--ranger_config_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:34507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.197.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:43:43.701694 12118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:43:43.703069 12118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:43:43.716818 12126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:43:43.716781 12124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:43:43.716868 12123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:43:43.717722 12118 server_base.cc:1034] running on GCE node
I20260929 16:43:43.718605 12118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:43:43.720278 12118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:43:43.721521 12118 hybrid_clock.cc:648] HybridClock initialized: now 1790700223721464 us; error 62 us; skew 500 ppm
I20260929 16:43:43.721896 12118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:43:43 dist-test-slave-l31f krb5kdc[11674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700223, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:43:43.734030 12118 init.cc:377] Logged in from keytab as kudu/127.0.197.126@KRBTEST.COM (short username kudu)
I20260929 16:43:43.735997 12118 webserver.cc:460] Webserver started at http://127.0.197.126:36093/ using document root <none> and password file <none>
I20260929 16:43:43.736711 12118 fs_manager.cc:359] Metadata directory not provided
I20260929 16:43:43.736815 12118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:43:43.737078 12118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:43:43 dist-test-slave-l31f krb5kdc[11674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700223, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for HTTP/127.0.197.84@KRBTEST.COM
I20260929 16:43:43.798658 12118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/master-0/data/instance:
uuid: "5f20953cdf7147259de298ae38796ebd"
format_stamp: "Formatted at 2026-09-29 16:43:43 on dist-test-slave-l31f"
server_key: "af6b461212f8fd17d9ed584cd55ebfb9"
server_key_iv: "ffd710926bdc23c268724df65e16cfaf"
server_key_version: "kuduclusterkey@0"
I20260929 16:43:43.799840 12118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/master-0/wal/instance:
uuid: "5f20953cdf7147259de298ae38796ebd"
format_stamp: "Formatted at 2026-09-29 16:43:43 on dist-test-slave-l31f"
server_key: "af6b461212f8fd17d9ed584cd55ebfb9"
server_key_iv: "ffd710926bdc23c268724df65e16cfaf"
server_key_version: "kuduclusterkey@0"
I20260929 16:43:43.807264 12118 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260929 16:43:43.829428 12133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:43:43.831302 12118 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:43:43.831521 12118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "5f20953cdf7147259de298ae38796ebd"
format_stamp: "Formatted at 2026-09-29 16:43:43 on dist-test-slave-l31f"
server_key: "af6b461212f8fd17d9ed584cd55ebfb9"
server_key_iv: "ffd710926bdc23c268724df65e16cfaf"
server_key_version: "kuduclusterkey@0"
I20260929 16:43:43.831758 12118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:43:43.849943 12118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:43:43.854434 12118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:43:43.854795 12118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:43:43.877732 12118 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.126:34507
I20260929 16:43:43.877744 12184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.126:34507 every 8 connection(s)
I20260929 16:43:43.879405 12118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:43:43.884109 12185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:43:43.888820 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 12118
I20260929 16:43:43.889087 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/master-0/wal/instance
I20260929 16:43:43.889770 789 external_mini_cluster.cc:1428] Setting key 85416c3838d2d73df3c77266ff749593
I20260929 16:43:43.895495 12185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f20953cdf7147259de298ae38796ebd: Bootstrap starting.
I20260929 16:43:43.899814 12185 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f20953cdf7147259de298ae38796ebd: Neither blocks nor log segments found. Creating new log.
I20260929 16:43:43.901139 12185 log.cc:824] T 00000000000000000000000000000000 P 5f20953cdf7147259de298ae38796ebd: Log is configured to *not* fsync() on all Append() calls
I20260929 16:43:43.904315 12185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f20953cdf7147259de298ae38796ebd: No bootstrap required, opened a new log
Sep 29 16:43:43 dist-test-slave-l31f krb5kdc[11674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:43:43.915584 12185 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f20953cdf7147259de298ae38796ebd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f20953cdf7147259de298ae38796ebd" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 34507 } }
I20260929 16:43:43.916059 12185 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f20953cdf7147259de298ae38796ebd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:43:43.916167 12185 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f20953cdf7147259de298ae38796ebd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f20953cdf7147259de298ae38796ebd, State: Initialized, Role: FOLLOWER
I20260929 16:43:43.916646 12188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:43:43.891980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60472 (local address 127.0.197.126:34507)
0929 16:43:43.892964 (+ 984us) server_negotiation.cc:206] Beginning negotiation
0929 16:43:43.892978 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:43:43.893029 (+ 51us) server_negotiation.cc:407] Connection header received
0929 16:43:43.894912 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:43:43.894938 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:43:43.895487 (+ 549us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:43:43.895922 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:43:43.897931 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:43:43.899493 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:43:43.901380 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:43:43.902240 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:43:43.902399 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:43:43.906686 (+ 4287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:43:43.906697 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:43:43.906720 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:43:43.906784 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:43:43.911425 (+ 4641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:43:43.912319 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:43:43.912328 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:43:43.912338 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:43:43.912560 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:43:43.913049 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:43:43.913054 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:43:43.913058 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:43:43.913905 (+ 847us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:43:43.914221 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0929 16:43:43.914723 (+ 502us) server_negotiation.cc:299] Negotiation successful
0929 16:43:43.915219 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":700,"thread_start_us":438,"threads_started":1}
I20260929 16:43:43.916913 12185 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f20953cdf7147259de298ae38796ebd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f20953cdf7147259de298ae38796ebd" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 34507 } }
I20260929 16:43:43.917218 12185 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f20953cdf7147259de298ae38796ebd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:43:43.917379 12185 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f20953cdf7147259de298ae38796ebd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:43:43.917579 12185 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f20953cdf7147259de298ae38796ebd [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:43:43.920390 12185 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f20953cdf7147259de298ae38796ebd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f20953cdf7147259de298ae38796ebd" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 34507 } }
I20260929 16:43:43.920888 12185 leader_election.cc:302] T 00000000000000000000000000000000 P 5f20953cdf7147259de298ae38796ebd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f20953cdf7147259de298ae38796ebd; no voters:
I20260929 16:43:43.921476 12185 leader_election.cc:288] T 00000000000000000000000000000000 P 5f20953cdf7147259de298ae38796ebd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:43:43.921718 12190 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f20953cdf7147259de298ae38796ebd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:43:43.923754 12190 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f20953cdf7147259de298ae38796ebd [term 1 LEADER]: Becoming Leader. State: Replica: 5f20953cdf7147259de298ae38796ebd, State: Running, Role: LEADER
I20260929 16:43:43.924760 12185 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f20953cdf7147259de298ae38796ebd [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:43:43.924635 12190 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f20953cdf7147259de298ae38796ebd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f20953cdf7147259de298ae38796ebd" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 34507 } }
I20260929 16:43:43.929361 12191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f20953cdf7147259de298ae38796ebd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f20953cdf7147259de298ae38796ebd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f20953cdf7147259de298ae38796ebd" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 34507 } } }
I20260929 16:43:43.929813 12191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f20953cdf7147259de298ae38796ebd [sys.catalog]: This master's current role is: LEADER
I20260929 16:43:43.930408 12192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f20953cdf7147259de298ae38796ebd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f20953cdf7147259de298ae38796ebd. Latest consensus state: current_term: 1 leader_uuid: "5f20953cdf7147259de298ae38796ebd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f20953cdf7147259de298ae38796ebd" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 34507 } } }
I20260929 16:43:43.930764 12192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f20953cdf7147259de298ae38796ebd [sys.catalog]: This master's current role is: LEADER
I20260929 16:43:43.937045 12185 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 16:43:43.943619 12200 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:43:43.951555 12200 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:43:43.963968 12200 catalog_manager.cc:1347] Generated new cluster ID: 855c5683326c4692b0c7cae68dc06d4c
I20260929 16:43:43.964107 12200 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:43:44.004770 12200 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:43:44.007454 12200 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:43:44.018165 12200 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f20953cdf7147259de298ae38796ebd: Generated new TSK 0
I20260929 16:43:44.019464 12200 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.197.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab.
Entry for principal kudu/127.0.197.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab.
I20260929 16:43:44.872071 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.197.84:56653/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.65:0
--local_ip_for_outbound_sockets=127.0.197.65
--webserver_interface=127.0.197.65
--webserver_port=0
--tserver_master_addrs=127.0.197.126:34507
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab
--principal=kudu/127.0.197.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:33399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:43:45 dist-test-slave-l31f krb5kdc[11674](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790700225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 16:43:45.378422 12230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:43:45.378961 12230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:43:45.379194 12230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:43:45.392021 12230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:43:45.392273 12230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:43:45.392652 12230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.65
I20260929 16:43:45.405668 12230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:33399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.197.84:56653/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab
--principal=kudu/127.0.197.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.197.65
--webserver_port=0
--tserver_master_addrs=127.0.197.126:34507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.65
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:43:45.407276 12230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:43:45.408738 12230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:43:45.428328 12241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:43:45.432771 12240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:43:45.434703 12243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:43:45.435595 12230 server_base.cc:1034] running on GCE node
I20260929 16:43:45.437194 12230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:43:45.447490 12230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:43:45.449047 12230 hybrid_clock.cc:648] HybridClock initialized: now 1790700225448905 us; error 150 us; skew 500 ppm
I20260929 16:43:45.449508 12230 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:43:45 dist-test-slave-l31f krb5kdc[11674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:43:45.453778 12230 init.cc:377] Logged in from keytab as kudu/127.0.197.65@KRBTEST.COM (short username kudu)
I20260929 16:43:45.461889 12230 webserver.cc:460] Webserver started at http://127.0.197.65:36629/ using document root <none> and password file <none>
I20260929 16:43:45.463830 12230 fs_manager.cc:359] Metadata directory not provided
I20260929 16:43:45.464098 12230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:43:45.464666 12230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:43:45 dist-test-slave-l31f krb5kdc[11674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.65@KRBTEST.COM for HTTP/127.0.197.84@KRBTEST.COM
I20260929 16:43:45.495816 12230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ts-0/data/instance:
uuid: "8770fcc1e6444c6194f1821cdad01667"
format_stamp: "Formatted at 2026-09-29 16:43:45 on dist-test-slave-l31f"
server_key: "542f9131916b57e0187bfd97efd5ed3f"
server_key_iv: "7075d9de16e3a66475d6dbfd4664ca32"
server_key_version: "kuduclusterkey@0"
I20260929 16:43:45.497341 12230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ts-0/wal/instance:
uuid: "8770fcc1e6444c6194f1821cdad01667"
format_stamp: "Formatted at 2026-09-29 16:43:45 on dist-test-slave-l31f"
server_key: "542f9131916b57e0187bfd97efd5ed3f"
server_key_iv: "7075d9de16e3a66475d6dbfd4664ca32"
server_key_version: "kuduclusterkey@0"
I20260929 16:43:45.511922 12230 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.008s sys 0.004s
I20260929 16:43:45.547780 12250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:43:45.556861 12230 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.006s sys 0.006s
I20260929 16:43:45.557123 12230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ts-0/wal
uuid: "8770fcc1e6444c6194f1821cdad01667"
format_stamp: "Formatted at 2026-09-29 16:43:45 on dist-test-slave-l31f"
server_key: "542f9131916b57e0187bfd97efd5ed3f"
server_key_iv: "7075d9de16e3a66475d6dbfd4664ca32"
server_key_version: "kuduclusterkey@0"
I20260929 16:43:45.557456 12230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:43:45.616165 12230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:43:45.624887 12230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:43:45.625278 12230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:43:45.627300 12230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:43:45.640128 12230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:43:45.640255 12230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:43:45.640379 12230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:43:45.640427 12230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:43:45 dist-test-slave-l31f krb5kdc[11674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for HTTP/127.0.197.84@KRBTEST.COM
I20260929 16:43:45.826439 12230 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.65:43139
I20260929 16:43:45.826498 12362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.65:43139 every 8 connection(s)
I20260929 16:43:45.828244 12230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
I20260929 16:43:45.829732 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 12230
I20260929 16:43:45.830009 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ts-0/wal/instance
I20260929 16:43:45.831159 789 external_mini_cluster.cc:1428] Setting key 7e05bb1bbb417dca3251d7bdc5ffc715
WARNING: no policy specified for kudu/127.0.197.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.66@KRBTEST.COM" created.
Sep 29 16:43:45 dist-test-slave-l31f krb5kdc[11674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.65@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:43:45.900696 12365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:43:45.836497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:33903 (local address 127.0.197.126:34507)
0929 16:43:45.846993 (+ 10496us) server_negotiation.cc:206] Beginning negotiation
0929 16:43:45.847001 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:43:45.850381 (+ 3380us) server_negotiation.cc:407] Connection header received
0929 16:43:45.866131 (+ 15750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:43:45.866141 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:43:45.866314 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:43:45.866478 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:43:45.870523 (+ 4045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:43:45.872518 (+ 1995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:43:45.875822 (+ 3304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:43:45.876509 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:43:45.876619 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:43:45.891805 (+ 15186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:43:45.891813 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:43:45.891821 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:43:45.891872 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:43:45.895752 (+ 3880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:43:45.896984 (+ 1232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:43:45.896991 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:43:45.896995 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:43:45.897160 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:43:45.897769 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:43:45.897774 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:43:45.897777 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:43:45.898604 (+ 827us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:43:45.898755 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 16:43:45.900008 (+ 1253us) server_negotiation.cc:299] Negotiation successful
0929 16:43:45.900307 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9839,"thread_start_us":9708,"threads_started":1}
I20260929 16:43:45.903209 12366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:43:45.837073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:34507 (local address 127.0.197.65:33903)
0929 16:43:45.849262 (+ 12189us) negotiation.cc:107] Waiting for socket to connect
0929 16:43:45.849328 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 16:43:45.865175 (+ 15847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:43:45.867379 (+ 2204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:43:45.867405 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:43:45.868478 (+ 1073us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:43:45.870065 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:43:45.870117 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:43:45.872952 (+ 2835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:43:45.872961 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:43:45.875356 (+ 2395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:43:45.875384 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:43:45.877177 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:43:45.877184 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:43:45.887213 (+ 10029us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:43:45.888335 (+ 1122us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:43:45.888378 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:43:45.891406 (+ 3028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:43:45.896014 (+ 4608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:43:45.896025 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:43:45.896048 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:43:45.896794 (+ 746us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:43:45.897329 (+ 535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:43:45.897334 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:43:45.897338 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:43:45.897602 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:43:45.898865 (+ 1263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:43:45.898878 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:43:45.899491 (+ 613us) client_negotiation.cc:769] Sending connection context
0929 16:43:45.901064 (+ 1573us) client_negotiation.cc:240] Negotiation successful
0929 16:43:45.901676 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11788,"thread_start_us":13207,"threads_started":1}
I20260929 16:43:45.905369 12363 heartbeater.cc:347] Connected to a master server at 127.0.197.126:34507
I20260929 16:43:45.905723 12363 heartbeater.cc:464] Registering TS with master...
I20260929 16:43:45.906623 12363 heartbeater.cc:511] Master 127.0.197.126:34507 requested a full tablet report, sending...
I20260929 16:43:45.909478 12150 ts_manager.cc:194] Registered new tserver with Master: 8770fcc1e6444c6194f1821cdad01667 (127.0.197.65:43139)
I20260929 16:43:45.911693 12150 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.65@KRBTEST.COM'} at 127.0.197.65:33903
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab.
Entry for principal kudu/127.0.197.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab.
W20260929 16:43:45.939550 789 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.197.66: real 0.105s user 0.002s sys 0.018s
I20260929 16:43:45.940347 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.197.84:56653/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.66:0
--local_ip_for_outbound_sockets=127.0.197.66
--webserver_interface=127.0.197.66
--webserver_port=0
--tserver_master_addrs=127.0.197.126:34507
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab
--principal=kudu/127.0.197.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:33399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:43:46 dist-test-slave-l31f krb5kdc[11674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for HTTP/127.0.197.84@KRBTEST.COM
W20260929 16:43:46.439442 12369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:43:46.439909 12369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:43:46.440088 12369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:43:46.450867 12369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:43:46.451161 12369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:43:46.451349 12369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.66
I20260929 16:43:46.463861 12369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:33399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.197.84:56653/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab
--principal=kudu/127.0.197.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.197.66
--webserver_port=0
--tserver_master_addrs=127.0.197.126:34507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.66
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:43:46.465456 12369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:43:46.466919 12369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:43:46.483806 12384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:43:46.483767 12383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:43:46.485129 12386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:43:46.485981 12369 server_base.cc:1034] running on GCE node
I20260929 16:43:46.486869 12369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:43:46.488235 12369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:43:46.489508 12369 hybrid_clock.cc:648] HybridClock initialized: now 1790700226489460 us; error 38 us; skew 500 ppm
I20260929 16:43:46.489946 12369 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:43:46 dist-test-slave-l31f krb5kdc[11674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700226, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:43:46.494678 12369 init.cc:377] Logged in from keytab as kudu/127.0.197.66@KRBTEST.COM (short username kudu)
I20260929 16:43:46.496990 12369 webserver.cc:460] Webserver started at http://127.0.197.66:36291/ using document root <none> and password file <none>
I20260929 16:43:46.498031 12369 fs_manager.cc:359] Metadata directory not provided
I20260929 16:43:46.498188 12369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:43:46.498548 12369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:43:46 dist-test-slave-l31f krb5kdc[11674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700226, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.66@KRBTEST.COM for HTTP/127.0.197.84@KRBTEST.COM
I20260929 16:43:46.520799 12369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ts-1/data/instance:
uuid: "e46634049cec42ccb0898b0992182e15"
format_stamp: "Formatted at 2026-09-29 16:43:46 on dist-test-slave-l31f"
server_key: "7d5f26ee098cb2b2bcc566b7aecb7427"
server_key_iv: "dbc360e02feb825b5847ffe56b79f81b"
server_key_version: "kuduclusterkey@0"
I20260929 16:43:46.522116 12369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ts-1/wal/instance:
uuid: "e46634049cec42ccb0898b0992182e15"
format_stamp: "Formatted at 2026-09-29 16:43:46 on dist-test-slave-l31f"
server_key: "7d5f26ee098cb2b2bcc566b7aecb7427"
server_key_iv: "dbc360e02feb825b5847ffe56b79f81b"
server_key_version: "kuduclusterkey@0"
I20260929 16:43:46.532671 12369 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.010s sys 0.001s
I20260929 16:43:46.556315 12393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:43:46.559217 12369 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.005s sys 0.000s
I20260929 16:43:46.559553 12369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ts-1/wal
uuid: "e46634049cec42ccb0898b0992182e15"
format_stamp: "Formatted at 2026-09-29 16:43:46 on dist-test-slave-l31f"
server_key: "7d5f26ee098cb2b2bcc566b7aecb7427"
server_key_iv: "dbc360e02feb825b5847ffe56b79f81b"
server_key_version: "kuduclusterkey@0"
I20260929 16:43:46.559937 12369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:43:46.587234 12369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:43:46.592409 12369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:43:46.592763 12369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:43:46.593900 12369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:43:46.595736 12369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:43:46.595857 12369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:43:46.596014 12369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:43:46.596086 12369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:43:46.647156 12369 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.66:38919
I20260929 16:43:46.647171 12505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.66:38919 every 8 connection(s)
I20260929 16:43:46.648653 12369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
I20260929 16:43:46.654130 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 12369
I20260929 16:43:46.654331 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ts-1/wal/instance
I20260929 16:43:46.654906 789 external_mini_cluster.cc:1428] Setting key 57750cc423a6989896ef4c9d84e15e0d
Sep 29 16:43:46 dist-test-slave-l31f krb5kdc[11674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700226, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.66@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:43:46.675637 12508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:43:46.652144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:56357 (local address 127.0.197.126:34507)
0929 16:43:46.652665 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0929 16:43:46.652671 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:43:46.654844 (+ 2173us) server_negotiation.cc:407] Connection header received
0929 16:43:46.659162 (+ 4318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:43:46.659167 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:43:46.659254 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:43:46.659405 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:43:46.661188 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:43:46.661946 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:43:46.663834 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:43:46.664260 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:43:46.664410 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:43:46.669158 (+ 4748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:43:46.669164 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:43:46.669167 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:43:46.669216 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:43:46.672106 (+ 2890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:43:46.673038 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:43:46.673043 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:43:46.673045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:43:46.673178 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:43:46.673690 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:43:46.673702 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:43:46.673704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:43:46.674009 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:43:46.674169 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 16:43:46.674946 (+ 777us) server_negotiation.cc:299] Negotiation successful
0929 16:43:46.675240 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":248,"threads_started":1}
I20260929 16:43:46.677664 12509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:43:46.652909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:34507 (local address 127.0.197.66:56357)
0929 16:43:46.654622 (+ 1713us) negotiation.cc:107] Waiting for socket to connect
0929 16:43:46.654682 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 16:43:46.657030 (+ 2348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:43:46.659739 (+ 2709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:43:46.659751 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:43:46.660286 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:43:46.660963 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:43:46.660985 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:43:46.662153 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:43:46.662158 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:43:46.663644 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:43:46.663660 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:43:46.664542 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:43:46.664550 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:43:46.664809 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:43:46.665608 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:43:46.665633 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:43:46.668835 (+ 3202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:43:46.672327 (+ 3492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:43:46.672338 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:43:46.672360 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:43:46.672869 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:43:46.673304 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:43:46.673310 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:43:46.673313 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:43:46.673563 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:43:46.674144 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:43:46.674153 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:43:46.674546 (+ 393us) client_negotiation.cc:769] Sending connection context
0929 16:43:46.674833 (+ 287us) client_negotiation.cc:240] Negotiation successful
0929 16:43:46.675477 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":362,"threads_started":1}
I20260929 16:43:46.679471 12506 heartbeater.cc:347] Connected to a master server at 127.0.197.126:34507
I20260929 16:43:46.679849 12506 heartbeater.cc:464] Registering TS with master...
I20260929 16:43:46.680828 12506 heartbeater.cc:511] Master 127.0.197.126:34507 requested a full tablet report, sending...
I20260929 16:43:46.682886 12150 ts_manager.cc:194] Registered new tserver with Master: e46634049cec42ccb0898b0992182e15 (127.0.197.66:38919)
I20260929 16:43:46.684127 12150 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.66@KRBTEST.COM'} at 127.0.197.66:56357
WARNING: no policy specified for kudu/127.0.197.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab.
Entry for principal kudu/127.0.197.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab.
I20260929 16:43:46.723134 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.197.84:56653/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.67:0
--local_ip_for_outbound_sockets=127.0.197.67
--webserver_interface=127.0.197.67
--webserver_port=0
--tserver_master_addrs=127.0.197.126:34507
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab
--principal=kudu/127.0.197.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:33399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:43:46.916209 12363 heartbeater.cc:503] Master 127.0.197.126:34507 was elected leader, sending a full tablet report...
W20260929 16:43:47.103920 12512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:43:47.104303 12512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:43:47.104874 12512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:43:47.113170 12512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:43:47.113395 12512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:43:47.113580 12512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.67
I20260929 16:43:47.124377 12512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:33399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.197.84:56653/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab
--principal=kudu/127.0.197.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.197.67
--webserver_port=0
--tserver_master_addrs=127.0.197.126:34507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.67
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:43:47.125869 12512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:43:47.127369 12512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:43:47.144083 12517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:43:47.144176 12518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:43:47.144212 12520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:43:47.145273 12512 server_base.cc:1034] running on GCE node
I20260929 16:43:47.146128 12512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:43:47.148212 12512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:43:47.149648 12512 hybrid_clock.cc:648] HybridClock initialized: now 1790700227149578 us; error 37 us; skew 500 ppm
I20260929 16:43:47.150393 12512 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:43:47 dist-test-slave-l31f krb5kdc[11674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:43:47.154130 12512 init.cc:377] Logged in from keytab as kudu/127.0.197.67@KRBTEST.COM (short username kudu)
I20260929 16:43:47.156035 12512 webserver.cc:460] Webserver started at http://127.0.197.67:40695/ using document root <none> and password file <none>
I20260929 16:43:47.156803 12512 fs_manager.cc:359] Metadata directory not provided
I20260929 16:43:47.156944 12512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:43:47.157292 12512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:43:47 dist-test-slave-l31f krb5kdc[11674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.67@KRBTEST.COM for HTTP/127.0.197.84@KRBTEST.COM
I20260929 16:43:47.173302 12512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ts-2/data/instance:
uuid: "8b14f1be1eaf44099beb64abf3e2b33d"
format_stamp: "Formatted at 2026-09-29 16:43:47 on dist-test-slave-l31f"
server_key: "85d94cafa9b1573c34cd246ad40310ce"
server_key_iv: "b675733cde7c8302015e8fd3c1f26552"
server_key_version: "kuduclusterkey@0"
I20260929 16:43:47.174269 12512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ts-2/wal/instance:
uuid: "8b14f1be1eaf44099beb64abf3e2b33d"
format_stamp: "Formatted at 2026-09-29 16:43:47 on dist-test-slave-l31f"
server_key: "85d94cafa9b1573c34cd246ad40310ce"
server_key_iv: "b675733cde7c8302015e8fd3c1f26552"
server_key_version: "kuduclusterkey@0"
I20260929 16:43:47.180117 12512 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260929 16:43:47.195812 12527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:43:47.197511 12512 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:43:47.197705 12512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ts-2/wal
uuid: "8b14f1be1eaf44099beb64abf3e2b33d"
format_stamp: "Formatted at 2026-09-29 16:43:47 on dist-test-slave-l31f"
server_key: "85d94cafa9b1573c34cd246ad40310ce"
server_key_iv: "b675733cde7c8302015e8fd3c1f26552"
server_key_version: "kuduclusterkey@0"
I20260929 16:43:47.197955 12512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:43:47.219462 12512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:43:47.224086 12512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:43:47.224363 12512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:43:47.225569 12512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:43:47.227342 12512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:43:47.227427 12512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:43:47.227519 12512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:43:47.227563 12512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:43:47.269460 12512 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.67:44389
I20260929 16:43:47.269569 12639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.67:44389 every 8 connection(s)
I20260929 16:43:47.270936 12512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
I20260929 16:43:47.281150 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 12512
I20260929 16:43:47.281355 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ts-2/wal/instance
I20260929 16:43:47.281816 789 external_mini_cluster.cc:1428] Setting key aff36685839b7d161ee70e40fe293ae4
Sep 29 16:43:47 dist-test-slave-l31f krb5kdc[11674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.67@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:43:47.293838 12642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:43:47.273887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:44891 (local address 127.0.197.126:34507)
0929 16:43:47.274499 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0929 16:43:47.274504 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:43:47.275415 (+ 911us) server_negotiation.cc:407] Connection header received
0929 16:43:47.277947 (+ 2532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:43:47.277954 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:43:47.278056 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:43:47.278185 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:43:47.279908 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:43:47.280692 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:43:47.282648 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:43:47.283487 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:43:47.283649 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:43:47.287607 (+ 3958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:43:47.287612 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:43:47.287616 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:43:47.287652 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:43:47.290300 (+ 2648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:43:47.291431 (+ 1131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:43:47.291436 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:43:47.291438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:43:47.291590 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:43:47.292118 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:43:47.292122 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:43:47.292124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:43:47.292421 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:43:47.292555 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 16:43:47.293371 (+ 816us) server_negotiation.cc:299] Negotiation successful
0929 16:43:47.293562 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":350,"threads_started":1}
I20260929 16:43:47.294860 12643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:43:47.274389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:34507 (local address 127.0.197.67:44891)
0929 16:43:47.275225 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0929 16:43:47.275271 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 16:43:47.277617 (+ 2346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:43:47.278486 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:43:47.278496 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:43:47.278963 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:43:47.279705 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:43:47.279724 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:43:47.280875 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:43:47.280879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:43:47.282427 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:43:47.282443 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:43:47.283771 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:43:47.283776 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:43:47.284065 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:43:47.284884 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:43:47.284909 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:43:47.287352 (+ 2443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:43:47.290487 (+ 3135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:43:47.290496 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:43:47.290512 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:43:47.291275 (+ 763us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:43:47.291702 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:43:47.291705 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:43:47.291707 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:43:47.291932 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:43:47.292570 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:43:47.292577 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:43:47.292947 (+ 370us) client_negotiation.cc:769] Sending connection context
0929 16:43:47.293208 (+ 261us) client_negotiation.cc:240] Negotiation successful
0929 16:43:47.293632 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":305,"threads_started":1}
I20260929 16:43:47.296640 12640 heartbeater.cc:347] Connected to a master server at 127.0.197.126:34507
I20260929 16:43:47.297022 12640 heartbeater.cc:464] Registering TS with master...
I20260929 16:43:47.298007 12640 heartbeater.cc:511] Master 127.0.197.126:34507 requested a full tablet report, sending...
I20260929 16:43:47.299608 12150 ts_manager.cc:194] Registered new tserver with Master: 8b14f1be1eaf44099beb64abf3e2b33d (127.0.197.67:44389)
I20260929 16:43:47.300410 12150 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.67@KRBTEST.COM'} at 127.0.197.67:44891
I20260929 16:43:47.310920 789 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:43:47.331727 12642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:43:47.316221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45250 (local address 127.0.197.126:34507)
0929 16:43:47.316419 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 16:43:47.316423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:43:47.316844 (+ 421us) server_negotiation.cc:407] Connection header received
0929 16:43:47.316982 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:43:47.316986 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:43:47.317071 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:43:47.317178 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:43:47.318727 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:43:47.319622 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:43:47.321242 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:43:47.321694 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:43:47.321803 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:43:47.324698 (+ 2895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:43:47.324709 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:43:47.324714 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:43:47.324765 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:43:47.327953 (+ 3188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:43:47.329043 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:43:47.329049 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:43:47.329051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:43:47.329280 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:43:47.329766 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:43:47.329769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:43:47.329771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:43:47.330082 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:43:47.330393 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0929 16:43:47.330763 (+ 370us) server_negotiation.cc:299] Negotiation successful
0929 16:43:47.331313 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 16:43:47.337446 12150 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:43:47.345221 12150 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:43:47.375753 12656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:43:47.360572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60572 (local address 127.0.197.65:43139)
0929 16:43:47.365681 (+ 5109us) server_negotiation.cc:206] Beginning negotiation
0929 16:43:47.365689 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:43:47.365722 (+ 33us) server_negotiation.cc:407] Connection header received
0929 16:43:47.365847 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:43:47.365859 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:43:47.366189 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:43:47.367038 (+ 849us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:43:47.367479 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:43:47.369092 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:43:47.372396 (+ 3304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:43:47.374471 (+ 2075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:43:47.374629 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:43:47.374882 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0929 16:43:47.375220 (+ 338us) server_negotiation.cc:299] Negotiation successful
0929 16:43:47.375408 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2121,"thread_start_us":1916,"threads_started":1}
I20260929 16:43:47.375856 12653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:43:47.359784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:43139 (local address 127.0.0.1:60572)
0929 16:43:47.361596 (+ 1812us) negotiation.cc:107] Waiting for socket to connect
0929 16:43:47.361637 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 16:43:47.361905 (+ 268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:43:47.366467 (+ 4562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:43:47.366474 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:43:47.366509 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:43:47.366825 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:43:47.366871 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:43:47.369297 (+ 2426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:43:47.369304 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:43:47.372174 (+ 2870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:43:47.372191 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:43:47.374701 (+ 2510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:43:47.374708 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:43:47.374926 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:43:47.374984 (+ 58us) client_negotiation.cc:769] Sending connection context
0929 16:43:47.375165 (+ 181us) client_negotiation.cc:240] Negotiation successful
0929 16:43:47.375395 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1567,"mutex_wait_us":39,"thread_start_us":1334,"threads_started":1}
I20260929 16:43:47.376523 12657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:43:47.359844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.67:44389 (local address 127.0.0.1:56006)
0929 16:43:47.366018 (+ 6174us) negotiation.cc:107] Waiting for socket to connect
0929 16:43:47.366042 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 16:43:47.366222 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:43:47.367474 (+ 1252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:43:47.367482 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:43:47.367519 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:43:47.368041 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:43:47.368056 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:43:47.370328 (+ 2272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:43:47.370334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:43:47.372522 (+ 2188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:43:47.372534 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:43:47.375102 (+ 2568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:43:47.375108 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:43:47.375311 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:43:47.375347 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 16:43:47.375435 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 16:43:47.375579 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6002,"mutex_wait_us":1,"thread_start_us":6076,"threads_started":1}
I20260929 16:43:47.377069 12655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:43:47.359890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56006 (local address 127.0.197.67:44389)
0929 16:43:47.365606 (+ 5716us) server_negotiation.cc:206] Beginning negotiation
0929 16:43:47.365617 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:43:47.366457 (+ 840us) server_negotiation.cc:407] Connection header received
0929 16:43:47.366588 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:43:47.366599 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:43:47.366952 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:43:47.367207 (+ 255us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:43:47.368521 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:43:47.370135 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:43:47.373113 (+ 2978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:43:47.374486 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:43:47.374629 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:43:47.374883 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0929 16:43:47.376313 (+ 1430us) server_negotiation.cc:299] Negotiation successful
0929 16:43:47.376493 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5482,"thread_start_us":5281,"threads_started":1}
I20260929 16:43:47.377250 12654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:43:47.362181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.66:38919 (local address 127.0.0.1:34646)
0929 16:43:47.366190 (+ 4009us) negotiation.cc:107] Waiting for socket to connect
0929 16:43:47.366213 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 16:43:47.366347 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:43:47.369762 (+ 3415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:43:47.369768 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:43:47.369797 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:43:47.370442 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:43:47.370459 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:43:47.372214 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:43:47.372220 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:43:47.374218 (+ 1998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:43:47.374230 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:43:47.376406 (+ 2176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:43:47.376417 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:43:47.376618 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:43:47.376665 (+ 47us) client_negotiation.cc:769] Sending connection context
0929 16:43:47.376788 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 16:43:47.376929 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3894,"thread_start_us":3412,"threads_started":1}
I20260929 16:43:47.377686 12658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:43:47.362497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34646 (local address 127.0.197.66:38919)
0929 16:43:47.369101 (+ 6604us) server_negotiation.cc:206] Beginning negotiation
0929 16:43:47.369111 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:43:47.369146 (+ 35us) server_negotiation.cc:407] Connection header received
0929 16:43:47.369301 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:43:47.369307 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:43:47.369533 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:43:47.369774 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:43:47.370665 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:43:47.371712 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:43:47.374401 (+ 2689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:43:47.375900 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:43:47.376043 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:43:47.376242 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0929 16:43:47.376843 (+ 601us) server_negotiation.cc:299] Negotiation successful
0929 16:43:47.377012 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6420,"thread_start_us":6315,"threads_started":1}
I20260929 16:43:47.380342 12298 tablet_service.cc:1467] Processing CreateTablet for tablet eee88ad0519542768d77b2a94e08551b (DEFAULT_TABLE table=test-table [id=7001068857934c0a9cc1943ae67d2530]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:43:47.380618 12441 tablet_service.cc:1467] Processing CreateTablet for tablet eee88ad0519542768d77b2a94e08551b (DEFAULT_TABLE table=test-table [id=7001068857934c0a9cc1943ae67d2530]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:43:47.381018 12575 tablet_service.cc:1467] Processing CreateTablet for tablet eee88ad0519542768d77b2a94e08551b (DEFAULT_TABLE table=test-table [id=7001068857934c0a9cc1943ae67d2530]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:43:47.381858 12441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eee88ad0519542768d77b2a94e08551b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:43:47.381925 12298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eee88ad0519542768d77b2a94e08551b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:43:47.382344 12575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eee88ad0519542768d77b2a94e08551b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:43:47.395169 12659 tablet_bootstrap.cc:492] T eee88ad0519542768d77b2a94e08551b P 8770fcc1e6444c6194f1821cdad01667: Bootstrap starting.
I20260929 16:43:47.398648 12659 tablet_bootstrap.cc:654] T eee88ad0519542768d77b2a94e08551b P 8770fcc1e6444c6194f1821cdad01667: Neither blocks nor log segments found. Creating new log.
I20260929 16:43:47.398725 12660 tablet_bootstrap.cc:492] T eee88ad0519542768d77b2a94e08551b P 8b14f1be1eaf44099beb64abf3e2b33d: Bootstrap starting.
I20260929 16:43:47.400063 12659 log.cc:824] T eee88ad0519542768d77b2a94e08551b P 8770fcc1e6444c6194f1821cdad01667: Log is configured to *not* fsync() on all Append() calls
I20260929 16:43:47.400308 12661 tablet_bootstrap.cc:492] T eee88ad0519542768d77b2a94e08551b P e46634049cec42ccb0898b0992182e15: Bootstrap starting.
I20260929 16:43:47.403048 12660 tablet_bootstrap.cc:654] T eee88ad0519542768d77b2a94e08551b P 8b14f1be1eaf44099beb64abf3e2b33d: Neither blocks nor log segments found. Creating new log.
I20260929 16:43:47.404408 12660 log.cc:824] T eee88ad0519542768d77b2a94e08551b P 8b14f1be1eaf44099beb64abf3e2b33d: Log is configured to *not* fsync() on all Append() calls
I20260929 16:43:47.404459 12661 tablet_bootstrap.cc:654] T eee88ad0519542768d77b2a94e08551b P e46634049cec42ccb0898b0992182e15: Neither blocks nor log segments found. Creating new log.
I20260929 16:43:47.405751 12661 log.cc:824] T eee88ad0519542768d77b2a94e08551b P e46634049cec42ccb0898b0992182e15: Log is configured to *not* fsync() on all Append() calls
I20260929 16:43:47.406384 12659 tablet_bootstrap.cc:492] T eee88ad0519542768d77b2a94e08551b P 8770fcc1e6444c6194f1821cdad01667: No bootstrap required, opened a new log
I20260929 16:43:47.406886 12659 ts_tablet_manager.cc:1397] T eee88ad0519542768d77b2a94e08551b P 8770fcc1e6444c6194f1821cdad01667: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.005s
I20260929 16:43:47.408368 12661 tablet_bootstrap.cc:492] T eee88ad0519542768d77b2a94e08551b P e46634049cec42ccb0898b0992182e15: No bootstrap required, opened a new log
I20260929 16:43:47.408402 12660 tablet_bootstrap.cc:492] T eee88ad0519542768d77b2a94e08551b P 8b14f1be1eaf44099beb64abf3e2b33d: No bootstrap required, opened a new log
I20260929 16:43:47.408681 12661 ts_tablet_manager.cc:1397] T eee88ad0519542768d77b2a94e08551b P e46634049cec42ccb0898b0992182e15: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 16:43:47.408689 12660 ts_tablet_manager.cc:1397] T eee88ad0519542768d77b2a94e08551b P 8b14f1be1eaf44099beb64abf3e2b33d: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260929 16:43:47.417232 12661 raft_consensus.cc:348] T eee88ad0519542768d77b2a94e08551b P e46634049cec42ccb0898b0992182e15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b14f1be1eaf44099beb64abf3e2b33d" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 44389 } } peers { permanent_uuid: "8770fcc1e6444c6194f1821cdad01667" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43139 } } peers { permanent_uuid: "e46634049cec42ccb0898b0992182e15" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 38919 } }
I20260929 16:43:47.417673 12661 raft_consensus.cc:374] T eee88ad0519542768d77b2a94e08551b P e46634049cec42ccb0898b0992182e15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:43:47.417770 12661 raft_consensus.cc:729] T eee88ad0519542768d77b2a94e08551b P e46634049cec42ccb0898b0992182e15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e46634049cec42ccb0898b0992182e15, State: Initialized, Role: FOLLOWER
I20260929 16:43:47.418462 12661 consensus_queue.cc:260] T eee88ad0519542768d77b2a94e08551b P e46634049cec42ccb0898b0992182e15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b14f1be1eaf44099beb64abf3e2b33d" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 44389 } } peers { permanent_uuid: "8770fcc1e6444c6194f1821cdad01667" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43139 } } peers { permanent_uuid: "e46634049cec42ccb0898b0992182e15" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 38919 } }
I20260929 16:43:47.419232 12659 raft_consensus.cc:348] T eee88ad0519542768d77b2a94e08551b P 8770fcc1e6444c6194f1821cdad01667 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b14f1be1eaf44099beb64abf3e2b33d" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 44389 } } peers { permanent_uuid: "8770fcc1e6444c6194f1821cdad01667" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43139 } } peers { permanent_uuid: "e46634049cec42ccb0898b0992182e15" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 38919 } }
I20260929 16:43:47.419688 12506 heartbeater.cc:503] Master 127.0.197.126:34507 was elected leader, sending a full tablet report...
I20260929 16:43:47.419716 12659 raft_consensus.cc:374] T eee88ad0519542768d77b2a94e08551b P 8770fcc1e6444c6194f1821cdad01667 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:43:47.419809 12659 raft_consensus.cc:729] T eee88ad0519542768d77b2a94e08551b P 8770fcc1e6444c6194f1821cdad01667 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8770fcc1e6444c6194f1821cdad01667, State: Initialized, Role: FOLLOWER
I20260929 16:43:47.420481 12659 consensus_queue.cc:260] T eee88ad0519542768d77b2a94e08551b P 8770fcc1e6444c6194f1821cdad01667 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b14f1be1eaf44099beb64abf3e2b33d" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 44389 } } peers { permanent_uuid: "8770fcc1e6444c6194f1821cdad01667" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43139 } } peers { permanent_uuid: "e46634049cec42ccb0898b0992182e15" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 38919 } }
I20260929 16:43:47.420382 12660 raft_consensus.cc:348] T eee88ad0519542768d77b2a94e08551b P 8b14f1be1eaf44099beb64abf3e2b33d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b14f1be1eaf44099beb64abf3e2b33d" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 44389 } } peers { permanent_uuid: "8770fcc1e6444c6194f1821cdad01667" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43139 } } peers { permanent_uuid: "e46634049cec42ccb0898b0992182e15" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 38919 } }
I20260929 16:43:47.420830 12660 raft_consensus.cc:374] T eee88ad0519542768d77b2a94e08551b P 8b14f1be1eaf44099beb64abf3e2b33d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:43:47.420939 12660 raft_consensus.cc:729] T eee88ad0519542768d77b2a94e08551b P 8b14f1be1eaf44099beb64abf3e2b33d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b14f1be1eaf44099beb64abf3e2b33d, State: Initialized, Role: FOLLOWER
I20260929 16:43:47.421743 12660 consensus_queue.cc:260] T eee88ad0519542768d77b2a94e08551b P 8b14f1be1eaf44099beb64abf3e2b33d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b14f1be1eaf44099beb64abf3e2b33d" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 44389 } } peers { permanent_uuid: "8770fcc1e6444c6194f1821cdad01667" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43139 } } peers { permanent_uuid: "e46634049cec42ccb0898b0992182e15" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 38919 } }
I20260929 16:43:47.422564 12661 ts_tablet_manager.cc:1428] T eee88ad0519542768d77b2a94e08551b P e46634049cec42ccb0898b0992182e15: Time spent starting tablet: real 0.014s user 0.003s sys 0.010s
I20260929 16:43:47.424772 12659 ts_tablet_manager.cc:1428] T eee88ad0519542768d77b2a94e08551b P 8770fcc1e6444c6194f1821cdad01667: Time spent starting tablet: real 0.018s user 0.014s sys 0.007s
I20260929 16:43:47.425990 12640 heartbeater.cc:503] Master 127.0.197.126:34507 was elected leader, sending a full tablet report...
I20260929 16:43:47.427137 12660 ts_tablet_manager.cc:1428] T eee88ad0519542768d77b2a94e08551b P 8b14f1be1eaf44099beb64abf3e2b33d: Time spent starting tablet: real 0.018s user 0.012s sys 0.007s
I20260929 16:43:47.427630 12665 raft_consensus.cc:482] T eee88ad0519542768d77b2a94e08551b P e46634049cec42ccb0898b0992182e15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:43:47.427985 12665 raft_consensus.cc:504] T eee88ad0519542768d77b2a94e08551b P e46634049cec42ccb0898b0992182e15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b14f1be1eaf44099beb64abf3e2b33d" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 44389 } } peers { permanent_uuid: "8770fcc1e6444c6194f1821cdad01667" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43139 } } peers { permanent_uuid: "e46634049cec42ccb0898b0992182e15" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 38919 } }
I20260929 16:43:47.430294 12665 leader_election.cc:288] T eee88ad0519542768d77b2a94e08551b P e46634049cec42ccb0898b0992182e15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b14f1be1eaf44099beb64abf3e2b33d (127.0.197.67:44389), 8770fcc1e6444c6194f1821cdad01667 (127.0.197.65:43139)
I20260929 16:43:47.438891 12655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:43:47.430779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:43585 (local address 127.0.197.67:44389)
0929 16:43:47.430998 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 16:43:47.431027 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0929 16:43:47.432682 (+ 1655us) server_negotiation.cc:407] Connection header received
0929 16:43:47.432919 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:43:47.432924 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:43:47.432993 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:43:47.433095 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:43:47.433664 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:43:47.434376 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:43:47.436752 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:43:47.437545 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:43:47.438043 (+ 498us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:43:47.438125 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 16:43:47.438473 (+ 348us) server_negotiation.cc:299] Negotiation successful
0929 16:43:47.438608 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 16:43:47.439046 12669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:43:47.430562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.67:44389 (local address 127.0.197.66:43585)
0929 16:43:47.432562 (+ 2000us) negotiation.cc:107] Waiting for socket to connect
0929 16:43:47.432581 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:43:47.432763 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:43:47.433114 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:43:47.433118 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:43:47.433208 (+ 90us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:43:47.433522 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:43:47.433530 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:43:47.434581 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:43:47.434587 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:43:47.436574 (+ 1987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:43:47.436591 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:43:47.438064 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:43:47.438071 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:43:47.438275 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:43:47.438300 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 16:43:47.438439 (+ 139us) client_negotiation.cc:240] Negotiation successful
0929 16:43:47.438585 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1877,"spinlock_wait_cycles":27520,"thread_start_us":2132,"threads_started":1}
I20260929 16:43:47.439062 12656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:43:47.431501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:50189 (local address 127.0.197.65:43139)
0929 16:43:47.431687 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 16:43:47.431693 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:43:47.433098 (+ 1405us) server_negotiation.cc:407] Connection header received
0929 16:43:47.433301 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:43:47.433310 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:43:47.433368 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:43:47.433473 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:43:47.433997 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:43:47.434928 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:43:47.437255 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:43:47.438034 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:43:47.438233 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:43:47.438312 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 16:43:47.438633 (+ 321us) server_negotiation.cc:299] Negotiation successful
0929 16:43:47.438771 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 16:43:47.439530 12668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:43:47.431160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:43139 (local address 127.0.197.66:50189)
0929 16:43:47.432985 (+ 1825us) negotiation.cc:107] Waiting for socket to connect
0929 16:43:47.432999 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:43:47.433161 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:43:47.433494 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:43:47.433498 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:43:47.433524 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:43:47.433847 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:43:47.433855 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:43:47.435214 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:43:47.435220 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:43:47.437050 (+ 1830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:43:47.437063 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:43:47.438289 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:43:47.438296 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:43:47.438472 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:43:47.438494 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 16:43:47.438604 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 16:43:47.438750 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1722,"thread_start_us":667,"threads_started":1}
I20260929 16:43:47.440158 12595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eee88ad0519542768d77b2a94e08551b" candidate_uuid: "e46634049cec42ccb0898b0992182e15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b14f1be1eaf44099beb64abf3e2b33d" is_pre_election: true
I20260929 16:43:47.440527 12318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eee88ad0519542768d77b2a94e08551b" candidate_uuid: "e46634049cec42ccb0898b0992182e15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8770fcc1e6444c6194f1821cdad01667" is_pre_election: true
I20260929 16:43:47.440670 12595 raft_consensus.cc:2454] T eee88ad0519542768d77b2a94e08551b P 8b14f1be1eaf44099beb64abf3e2b33d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e46634049cec42ccb0898b0992182e15 in term 0.
I20260929 16:43:47.441016 12318 raft_consensus.cc:2454] T eee88ad0519542768d77b2a94e08551b P 8770fcc1e6444c6194f1821cdad01667 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e46634049cec42ccb0898b0992182e15 in term 0.
I20260929 16:43:47.441865 12397 leader_election.cc:302] T eee88ad0519542768d77b2a94e08551b P e46634049cec42ccb0898b0992182e15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b14f1be1eaf44099beb64abf3e2b33d, e46634049cec42ccb0898b0992182e15; no voters:
I20260929 16:43:47.442332 12665 raft_consensus.cc:2781] T eee88ad0519542768d77b2a94e08551b P e46634049cec42ccb0898b0992182e15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:43:47.442497 12665 raft_consensus.cc:482] T eee88ad0519542768d77b2a94e08551b P e46634049cec42ccb0898b0992182e15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:43:47.442605 12665 raft_consensus.cc:3037] T eee88ad0519542768d77b2a94e08551b P e46634049cec42ccb0898b0992182e15 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:43:47.445019 12665 raft_consensus.cc:504] T eee88ad0519542768d77b2a94e08551b P e46634049cec42ccb0898b0992182e15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b14f1be1eaf44099beb64abf3e2b33d" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 44389 } } peers { permanent_uuid: "8770fcc1e6444c6194f1821cdad01667" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43139 } } peers { permanent_uuid: "e46634049cec42ccb0898b0992182e15" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 38919 } }
I20260929 16:43:47.445767 12665 leader_election.cc:288] T eee88ad0519542768d77b2a94e08551b P e46634049cec42ccb0898b0992182e15 [CANDIDATE]: Term 1 election: Requested vote from peers 8b14f1be1eaf44099beb64abf3e2b33d (127.0.197.67:44389), 8770fcc1e6444c6194f1821cdad01667 (127.0.197.65:43139)
I20260929 16:43:47.446312 12595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eee88ad0519542768d77b2a94e08551b" candidate_uuid: "e46634049cec42ccb0898b0992182e15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b14f1be1eaf44099beb64abf3e2b33d"
I20260929 16:43:47.446409 12318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eee88ad0519542768d77b2a94e08551b" candidate_uuid: "e46634049cec42ccb0898b0992182e15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8770fcc1e6444c6194f1821cdad01667"
I20260929 16:43:47.446545 12595 raft_consensus.cc:3037] T eee88ad0519542768d77b2a94e08551b P 8b14f1be1eaf44099beb64abf3e2b33d [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:43:47.446688 12318 raft_consensus.cc:3037] T eee88ad0519542768d77b2a94e08551b P 8770fcc1e6444c6194f1821cdad01667 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:43:47.449021 12595 raft_consensus.cc:2454] T eee88ad0519542768d77b2a94e08551b P 8b14f1be1eaf44099beb64abf3e2b33d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e46634049cec42ccb0898b0992182e15 in term 1.
I20260929 16:43:47.449023 12318 raft_consensus.cc:2454] T eee88ad0519542768d77b2a94e08551b P 8770fcc1e6444c6194f1821cdad01667 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e46634049cec42ccb0898b0992182e15 in term 1.
I20260929 16:43:47.449546 12396 leader_election.cc:302] T eee88ad0519542768d77b2a94e08551b P e46634049cec42ccb0898b0992182e15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8770fcc1e6444c6194f1821cdad01667, e46634049cec42ccb0898b0992182e15; no voters:
I20260929 16:43:47.449970 12665 raft_consensus.cc:2781] T eee88ad0519542768d77b2a94e08551b P e46634049cec42ccb0898b0992182e15 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:43:47.450533 12665 raft_consensus.cc:686] T eee88ad0519542768d77b2a94e08551b P e46634049cec42ccb0898b0992182e15 [term 1 LEADER]: Becoming Leader. State: Replica: e46634049cec42ccb0898b0992182e15, State: Running, Role: LEADER
I20260929 16:43:47.451117 12665 consensus_queue.cc:237] T eee88ad0519542768d77b2a94e08551b P e46634049cec42ccb0898b0992182e15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b14f1be1eaf44099beb64abf3e2b33d" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 44389 } } peers { permanent_uuid: "8770fcc1e6444c6194f1821cdad01667" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43139 } } peers { permanent_uuid: "e46634049cec42ccb0898b0992182e15" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 38919 } }
I20260929 16:43:47.456364 12148 catalog_manager.cc:5495] T eee88ad0519542768d77b2a94e08551b P e46634049cec42ccb0898b0992182e15 reported cstate change: term changed from 0 to 1, leader changed from <none> to e46634049cec42ccb0898b0992182e15 (127.0.197.66). New cstate: current_term: 1 leader_uuid: "e46634049cec42ccb0898b0992182e15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b14f1be1eaf44099beb64abf3e2b33d" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 44389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8770fcc1e6444c6194f1821cdad01667" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e46634049cec42ccb0898b0992182e15" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 38919 } health_report { overall_health: HEALTHY } } }
I20260929 16:43:47.478708 12658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:43:47.472362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34652 (local address 127.0.197.66:38919)
0929 16:43:47.472591 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 16:43:47.472596 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:43:47.472615 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:43:47.472929 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:43:47.472935 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:43:47.473051 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:43:47.473274 (+ 223us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:43:47.473790 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:43:47.474932 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:43:47.476511 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:43:47.476927 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:43:47.477029 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:43:47.477576 (+ 547us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:43:47.478128 (+ 552us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:43:47.478287 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 16:43:47.478412 (+ 125us) server_negotiation.cc:299] Negotiation successful
0929 16:43:47.478486 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 16:43:47.488075 12595 raft_consensus.cc:1261] T eee88ad0519542768d77b2a94e08551b P 8b14f1be1eaf44099beb64abf3e2b33d [term 1 FOLLOWER]: Refusing update from remote peer e46634049cec42ccb0898b0992182e15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:43:47.488077 12318 raft_consensus.cc:1261] T eee88ad0519542768d77b2a94e08551b P 8770fcc1e6444c6194f1821cdad01667 [term 1 FOLLOWER]: Refusing update from remote peer e46634049cec42ccb0898b0992182e15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:43:47.489291 12665 consensus_queue.cc:1035] T eee88ad0519542768d77b2a94e08551b P e46634049cec42ccb0898b0992182e15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b14f1be1eaf44099beb64abf3e2b33d" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 44389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:43:47.489698 12670 consensus_queue.cc:1035] T eee88ad0519542768d77b2a94e08551b P e46634049cec42ccb0898b0992182e15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8770fcc1e6444c6194f1821cdad01667" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:43:47.508481 12675 mvcc.cc:205] Tried to move back new op lower bound from 7334708131778306048 to 7334708131643981824. Current Snapshot: MvccSnapshot[applied={T|T < 7334708131778306048}]
I20260929 16:43:47.520462 12148 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:43:47.520785 12148 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:43:47.524852 12148 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5f20953cdf7147259de298ae38796ebd: Sending DeleteTablet for 3 replicas of tablet eee88ad0519542768d77b2a94e08551b
I20260929 16:43:47.526292 12575 tablet_service.cc:1514] Processing DeleteTablet for tablet eee88ad0519542768d77b2a94e08551b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:43:47 UTC) from {username='kudu', principal='kudu/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:56006
I20260929 16:43:47.526495 12441 tablet_service.cc:1514] Processing DeleteTablet for tablet eee88ad0519542768d77b2a94e08551b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:43:47 UTC) from {username='kudu', principal='kudu/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:34646
I20260929 16:43:47.526624 12298 tablet_service.cc:1514] Processing DeleteTablet for tablet eee88ad0519542768d77b2a94e08551b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:43:47 UTC) from {username='kudu', principal='kudu/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:60572
I20260929 16:43:47.528863 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 12230
I20260929 16:43:47.546593 12684 tablet_replica.cc:330] T eee88ad0519542768d77b2a94e08551b P e46634049cec42ccb0898b0992182e15: stopping tablet replica
I20260929 16:43:47.550000 12684 raft_consensus.cc:2229] T eee88ad0519542768d77b2a94e08551b P e46634049cec42ccb0898b0992182e15 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:43:47.551553 12682 tablet_replica.cc:330] T eee88ad0519542768d77b2a94e08551b P 8b14f1be1eaf44099beb64abf3e2b33d: stopping tablet replica
I20260929 16:43:47.552105 12682 raft_consensus.cc:2229] T eee88ad0519542768d77b2a94e08551b P 8b14f1be1eaf44099beb64abf3e2b33d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:43:47.552723 12682 raft_consensus.cc:2258] T eee88ad0519542768d77b2a94e08551b P 8b14f1be1eaf44099beb64abf3e2b33d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:43:47.553742 12683 tablet_replica.cc:330] T eee88ad0519542768d77b2a94e08551b P 8770fcc1e6444c6194f1821cdad01667: stopping tablet replica
I20260929 16:43:47.554167 12683 raft_consensus.cc:2229] T eee88ad0519542768d77b2a94e08551b P 8770fcc1e6444c6194f1821cdad01667 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:43:47.554499 12684 raft_consensus.cc:2258] T eee88ad0519542768d77b2a94e08551b P e46634049cec42ccb0898b0992182e15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:43:47.554710 12683 raft_consensus.cc:2258] T eee88ad0519542768d77b2a94e08551b P 8770fcc1e6444c6194f1821cdad01667 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:43:47.557534 12684 ts_tablet_manager.cc:1905] T eee88ad0519542768d77b2a94e08551b P e46634049cec42ccb0898b0992182e15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:43:47.557523 12683 ts_tablet_manager.cc:1905] T eee88ad0519542768d77b2a94e08551b P 8770fcc1e6444c6194f1821cdad01667: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 16:43:47 dist-test-slave-l31f krb5kdc[11674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.65@KRBTEST.COM
I20260929 16:43:47.562840 12682 ts_tablet_manager.cc:1905] T eee88ad0519542768d77b2a94e08551b P 8b14f1be1eaf44099beb64abf3e2b33d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:43:47.566674 12683 ts_tablet_manager.cc:1918] T eee88ad0519542768d77b2a94e08551b P 8770fcc1e6444c6194f1821cdad01667: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:43:47.566872 12683 log.cc:1196] T eee88ad0519542768d77b2a94e08551b P 8770fcc1e6444c6194f1821cdad01667: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ts-0/wal/wals/eee88ad0519542768d77b2a94e08551b
I20260929 16:43:47.567509 12683 ts_tablet_manager.cc:1939] T eee88ad0519542768d77b2a94e08551b P 8770fcc1e6444c6194f1821cdad01667: Deleting consensus metadata
I20260929 16:43:47.569819 12136 catalog_manager.cc:4841] TS 8770fcc1e6444c6194f1821cdad01667 (127.0.197.65:43139): tablet eee88ad0519542768d77b2a94e08551b (table test-table [id=7001068857934c0a9cc1943ae67d2530]) successfully deleted
I20260929 16:43:47.570438 12656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:43:47.530995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60578 (local address 127.0.197.65:43139)
0929 16:43:47.531294 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0929 16:43:47.531305 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:43:47.542365 (+ 11060us) server_negotiation.cc:407] Connection header received
0929 16:43:47.543113 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:43:47.543124 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:43:47.543257 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:43:47.543376 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:43:47.547224 (+ 3848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:43:47.548588 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:43:47.553208 (+ 4620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:43:47.554109 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:43:47.554289 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:43:47.560334 (+ 6045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:43:47.560354 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:43:47.560375 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:43:47.560428 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:43:47.565816 (+ 5388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:43:47.567148 (+ 1332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:43:47.567159 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:43:47.567163 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:43:47.567358 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:43:47.568035 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:43:47.568041 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:43:47.568044 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:43:47.568582 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:43:47.568936 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0929 16:43:47.569610 (+ 674us) server_negotiation.cc:299] Negotiation successful
0929 16:43:47.569885 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260929 16:43:47.572163 12358 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:43:47.576340 12684 ts_tablet_manager.cc:1918] T eee88ad0519542768d77b2a94e08551b P e46634049cec42ccb0898b0992182e15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:43:47.576583 12684 log.cc:1196] T eee88ad0519542768d77b2a94e08551b P e46634049cec42ccb0898b0992182e15: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ts-1/wal/wals/eee88ad0519542768d77b2a94e08551b
I20260929 16:43:47.577102 12684 ts_tablet_manager.cc:1939] T eee88ad0519542768d77b2a94e08551b P e46634049cec42ccb0898b0992182e15: Deleting consensus metadata
I20260929 16:43:47.578840 12136 catalog_manager.cc:4841] TS e46634049cec42ccb0898b0992182e15 (127.0.197.66:38919): tablet eee88ad0519542768d77b2a94e08551b (table test-table [id=7001068857934c0a9cc1943ae67d2530]) successfully deleted
I20260929 16:43:47.580725 12682 ts_tablet_manager.cc:1918] T eee88ad0519542768d77b2a94e08551b P 8b14f1be1eaf44099beb64abf3e2b33d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:43:47.580931 12682 log.cc:1196] T eee88ad0519542768d77b2a94e08551b P 8b14f1be1eaf44099beb64abf3e2b33d: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790700068255611-789-0/minicluster-data/ts-2/wal/wals/eee88ad0519542768d77b2a94e08551b
I20260929 16:43:47.581404 12682 ts_tablet_manager.cc:1939] T eee88ad0519542768d77b2a94e08551b P 8b14f1be1eaf44099beb64abf3e2b33d: Deleting consensus metadata
I20260929 16:43:47.583258 12137 catalog_manager.cc:4841] TS 8b14f1be1eaf44099beb64abf3e2b33d (127.0.197.67:44389): tablet eee88ad0519542768d77b2a94e08551b (table test-table [id=7001068857934c0a9cc1943ae67d2530]) successfully deleted
I20260929 16:43:47.711803 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 12230
I20260929 16:43:47.730175 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 12369
Sep 29 16:43:47 dist-test-slave-l31f krb5kdc[11674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.66@KRBTEST.COM
I20260929 16:43:47.748593 12658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:43:47.730991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34666 (local address 127.0.197.66:38919)
0929 16:43:47.731244 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0929 16:43:47.731250 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:43:47.731548 (+ 298us) server_negotiation.cc:407] Connection header received
0929 16:43:47.731757 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:43:47.731763 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:43:47.731861 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:43:47.731988 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:43:47.733262 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:43:47.734103 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:43:47.735619 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:43:47.736168 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:43:47.736343 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:43:47.741791 (+ 5448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:43:47.741810 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:43:47.741827 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:43:47.741879 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:43:47.745301 (+ 3422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:43:47.746187 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:43:47.746198 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:43:47.746200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:43:47.746367 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:43:47.746791 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:43:47.746800 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:43:47.746806 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:43:47.747302 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:43:47.747651 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0929 16:43:47.747978 (+ 327us) server_negotiation.cc:299] Negotiation successful
0929 16:43:47.748219 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260929 16:43:47.749352 12501 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:43:47.870095 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 12369
I20260929 16:43:47.889394 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 12512
Sep 29 16:43:47 dist-test-slave-l31f krb5kdc[11674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.67@KRBTEST.COM
I20260929 16:43:47.908143 12655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:43:47.890232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56018 (local address 127.0.197.67:44389)
0929 16:43:47.890420 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 16:43:47.890424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:43:47.890482 (+ 58us) server_negotiation.cc:407] Connection header received
0929 16:43:47.890697 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:43:47.890701 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:43:47.890767 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:43:47.890853 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:43:47.892755 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:43:47.893550 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:43:47.895622 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:43:47.896045 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:43:47.896175 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:43:47.901209 (+ 5034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:43:47.901227 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:43:47.901244 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:43:47.901294 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:43:47.904923 (+ 3629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:43:47.905830 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:43:47.905847 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:43:47.905850 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:43:47.906002 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:43:47.906419 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:43:47.906423 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:43:47.906425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:43:47.906886 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:43:47.907314 (+ 428us) server_negotiation.cc:1035] Waiting for connection context
0929 16:43:47.907551 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 16:43:47.907758 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 16:43:47.908931 12635 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:43:48.004554 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 12512
I20260929 16:43:48.022944 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 12118
I20260929 16:43:48.024472 12180 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:43:48.141795 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 12118
I20260929 16:43:48.169271 789 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 16:43:48.506304 789 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 16:43:48.506539 789 mini_ranger.cc:66] Stopping Ranger...
I20260929 16:43:48.853674 789 mini_ranger.cc:68] Stopped Ranger
2026-09-29 16:43:48.854 UTC [11704] LOG: received smart shutdown request
2026-09-29 16:43:48.858 UTC [11704] LOG: background worker "logical replication launcher" (PID 11713) exited with exit code 1
2026-09-29 16:43:48.858 UTC [11708] LOG: shutting down
2026-09-29 16:43:49.081 UTC [11704] LOG: database system is shut down
2026-09-29T16:43:49Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (53908 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:43:49 dist-test-slave-l31f krb5kdc[12698](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:43:49 dist-test-slave-l31f krb5kdc[12698](info): set up 2 sockets
Sep 29 16:43:49 dist-test-slave-l31f krb5kdc[12698](info): commencing operation
krb5kdc: starting...
W20260929 16:43:50.343796 789 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/test-user.keytab.
Sep 29 16:43:50 dist-test-slave-l31f krb5kdc[12698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:43:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:43:50Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.197.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.197.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.197.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/rangeradmin_127.0.197.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.197.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/rangeradmin_127.0.197.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.197.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.197.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.197.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/rangerlookup_127.0.197.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.197.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/rangerlookup_127.0.197.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.197.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.197.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.197.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/HTTP_127.0.197.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.197.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/HTTP_127.0.197.84@KRBTEST.COM.keytab.
I20260929 16:43:50.693851 789 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskQmsv3Z/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/postgres -l logfile start
2026-09-29 16:43:53.724 UTC [12728] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 16:43:53.724 UTC [12728] LOG: listening on IPv4 address "127.0.197.84", port 34827
2026-09-29 16:43:53.729 UTC [12728] LOG: listening on Unix socket "/tmp/.s.PGSQL.34827"
2026-09-29 16:43:53.752 UTC [12731] LOG: database system was shut down at 2026-09-29 16:43:51 UTC
2026-09-29 16:43:53.756 UTC [12728] LOG: database system is ready to accept connections
I20260929 16:43:54.739581 789 mini_postgres.cc:96] Postgres bound to 34827
2026-09-29 16:43:54.757 UTC [12739] FATAL: database "slave" does not exist
127.0.197.84:34827 - accepting connections
I20260929 16:43:54.758894 789 mini_ranger.cc:161] Starting Ranger...
I20260929 16:43:54.788108 789 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 16:43:55.461352 789 mini_ranger.cc:87] Created ranger Postgres database
I20260929 16:43:55.461483 789 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ranger-admin
2026-09-29 16:43:55,779 [I] DB FLAVOR :POSTGRES
2026-09-29 16:43:55,780 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 16:43:55,780 [I] Checking connection..
2026-09-29 16:43:55,780 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQmsv3Z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQmsv3Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.197.84:34827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 16:43:56,070 [I] Checking connection passed.
2026-09-29 16:43:56,070 [I] --------- Verifying version history table ---------
2026-09-29 16:43:56,070 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQmsv3Z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQmsv3Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.197.84:34827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 16:43:56,355 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 16:43:56,356 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQmsv3Z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQmsv3Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.197.84:34827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 16:43:56,625 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 16:43:56,626 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 16:43:56,626 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQmsv3Z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQmsv3Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.197.84:34827/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQmsv3Z/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 16:43:56.880 UTC [12803] WARNING: there is no transaction in progress
2026-09-29 16:43:56,887 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 16:43:56,887 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQmsv3Z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQmsv3Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.197.84:34827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 16:43:57,158 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 16:43:57,159 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 16:43:57,159 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQmsv3Z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQmsv3Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.197.84:34827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 16:43:57,444 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQmsv3Z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQmsv3Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.197.84:34827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 16:43:57,704 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQmsv3Z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQmsv3Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.197.84:34827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-l31f.c.gcp-upstream.internal','N') ;"
2026-09-29 16:43:57,977 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQmsv3Z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQmsv3Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.197.84:34827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 16:43:58,242 [I] Table x_portal_user does not exist in database ranger
2026-09-29 16:43:58,243 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQmsv3Z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQmsv3Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.197.84:34827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 16:43:58,497 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 16:43:58,498 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQmsv3Z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQmsv3Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.197.84:34827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 16:43:58,744 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQmsv3Z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQmsv3Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.197.84:34827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 16:43:59,022 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 16:43:59,022 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQmsv3Z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQmsv3Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.197.84:34827/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQmsv3Z/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 16:44:00.142 UTC [12911] WARNING: there is no transaction in progress
2026-09-29 16:44:00.163 UTC [12911] WARNING: there is no transaction in progress
2026-09-29 16:44:00.185 UTC [12911] WARNING: there is no transaction in progress
2026-09-29 16:44:00.206 UTC [12911] WARNING: there is no transaction in progress
2026-09-29 16:44:00.227 UTC [12911] WARNING: there is no transaction in progress
2026-09-29 16:44:00.249 UTC [12911] WARNING: there is no transaction in progress
2026-09-29 16:44:00.317 UTC [12911] WARNING: there is no transaction in progress
2026-09-29 16:44:00.327 UTC [12911] WARNING: there is no transaction in progress
2026-09-29 16:44:00.347 UTC [12911] WARNING: there is no transaction in progress
2026-09-29 16:44:00.360 UTC [12911] WARNING: there is no transaction in progress
2026-09-29 16:44:00.373 UTC [12911] WARNING: there is no transaction in progress
2026-09-29 16:44:00.390 UTC [12911] WARNING: there is no transaction in progress
2026-09-29 16:44:00.403 UTC [12911] WARNING: there is no transaction in progress
2026-09-29 16:44:00.412 UTC [12911] WARNING: there is no transaction in progress
2026-09-29 16:44:00.421 UTC [12911] WARNING: there is no transaction in progress
2026-09-29 16:44:00.428 UTC [12911] WARNING: there is no transaction in progress
2026-09-29 16:44:01,298 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 16:44:01,299 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQmsv3Z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQmsv3Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.197.84:34827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 16:44:01,547 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 16:44:01,547 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQmsv3Z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQmsv3Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.197.84:34827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 16:44:01,837 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 16:44:01,837 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQmsv3Z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQmsv3Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.197.84:34827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 16:44:02,103 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 16:44:02,103 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQmsv3Z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQmsv3Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.197.84:34827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 16:44:02,375 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQmsv3Z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQmsv3Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.197.84:34827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 16:44:02,668 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQmsv3Z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQmsv3Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.197.84:34827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-l31f.c.gcp-upstream.internal';"
2026-09-29 16:44:02,948 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 16:44:02,949 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQmsv3Z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQmsv3Z/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.197.84:34827/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 16:44:03,230 [I] DB_PATCHES have already been applied
I20260929 16:44:03.237032 789 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ranger-admin:/tmp/dist-test-taskQmsv3Z/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskQmsv3Z/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskQmsv3Z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQmsv3Z/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 16:44:03.237135 789 mini_ranger.cc:193] Using host: 127.0.197.84
I20260929 16:44:03.252367 789 mini_ranger.cc:227] Ranger admin URL: http://127.0.197.84:53561
Sep 29, 2026 4:44:03 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 4:44:03 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 4:44:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 4:44:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 4:44:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 4:44:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 4:44:04 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 16:44:04.417985 789 mini_ranger.cc:160] Time spent starting Ranger: real 9.659s user 0.007s sys 0.036s
Sep 29, 2026 4:44:27 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 4:44:27 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 4:44:27 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 4:44:27 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 4:44:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 4:44:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 4:44:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 4:44:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 4:44:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 4:44:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 4:44:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 4:44:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 4:44:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 4:44:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 4:44:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 4:44:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 4:44:27 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 4:44:28 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 16:44:29.082505 789 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.197.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.197.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.197.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/rangerkms_127.0.197.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.197.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/rangerkms_127.0.197.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.197.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.197.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.197.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/HTTP_127.0.197.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.197.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/HTTP_127.0.197.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 16:44:29 dist-test-slave-l31f krb5kdc[12698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700269, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 16:44:29.272261 789 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 16:44:29.295315 789 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 16:44:30.436177 789 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 16:44:30.436280 789 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ranger-kms
I20260929 16:44:30.436312 789 mini_ranger_kms.cc:227] Using postgres at 127.0.197.84:34827
2026-09-29 16:44:30,803 [I] DB FLAVOR :POSTGRES
2026-09-29 16:44:30,803 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 16:44:30,803 [I] Checking connection
2026-09-29 16:44:30,803 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQmsv3Z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.197.84:34827/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 16:44:31,055 [I] connection success
2026-09-29 16:44:31,056 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 16:44:31,056 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 16:44:31,056 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQmsv3Z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.197.84:34827/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 16:44:31,307 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 16:44:31,308 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 16:44:31,308 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 16:44:31,308 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQmsv3Z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.197.84:34827/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 16:44:31,615 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 16:44:32.187124 789 mini_ranger_kms.cc:310] Created kms service
I20260929 16:44:32.419360 789 mini_ranger_kms.cc:326] Created kudu user
I20260929 16:44:32.474131 789 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 16:44:32 dist-test-slave-l31f krb5kdc[12698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700269, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.197.84@KRBTEST.COM
I20260929 16:44:32.927544 789 mini_ranger_kms.cc:384] Added ranger policy
I20260929 16:44:32.927901 789 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ranger-kms:/tmp/dist-test-taskQmsv3Z/build/asan/bin/postgresql.jar:/tmp/dist-test-taskQmsv3Z/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskQmsv3Z/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 16:44:32.927981 789 mini_ranger_kms.cc:240] Using host: 127.0.197.84
I20260929 16:44:32.944720 789 mini_ranger_kms.cc:276] Ranger KMS PID: 13092
I20260929 16:44:32.944864 789 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.197.84:59275
Sep 29, 2026 4:44:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 4:44:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 4:44:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 4:44:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 4:44:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 4:44:34 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 16:44:35.253966 789 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.982s user 0.064s sys 0.318s
I20260929 16:44:35.254233 789 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 16:44:35.254422 789 mini_ranger_kms.cc:401] 127.0.197.84:59275/kms/v1/keys
Sep 29 16:44:35 dist-test-slave-l31f krb5kdc[12698](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790700275, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.197.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 16:44:35 dist-test-slave-l31f krb5kdc[12698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700275, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.197.84@KRBTEST.COM for HTTP/127.0.197.84@KRBTEST.COM
Sep 29 16:44:36 dist-test-slave-l31f krb5kdc[12698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700275, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.197.84@KRBTEST.COM for HTTP/127.0.197.84@KRBTEST.COM
Sep 29 16:44:38 dist-test-slave-l31f krb5kdc[12698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.197.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
I20260929 16:44:38.767971 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:39101
--webserver_interface=127.0.197.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:36181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.126:39101
--ranger_config_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.197.84:59275/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:44:39.132671 13142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:44:39.132994 13142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:44:39.133082 13142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:44:39.140473 13142 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:44:39.140605 13142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:44:39.140650 13142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:44:39.140714 13142 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:44:39.140762 13142 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:44:39.151294 13142 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:36181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.197.84:59275/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.197.126:39101
--ranger_config_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:39101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.197.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:44:39.152642 13142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:44:39.154022 13142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:44:39.168823 13147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:44:39.168938 13148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:44:39.169278 13142 server_base.cc:1034] running on GCE node
W20260929 16:44:39.168862 13150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:44:39.170513 13142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:44:39.172185 13142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:44:39.173458 13142 hybrid_clock.cc:648] HybridClock initialized: now 1790700279173395 us; error 60 us; skew 500 ppm
I20260929 16:44:39.173952 13142 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:44:39 dist-test-slave-l31f krb5kdc[12698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700279, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:44:39.177677 13142 init.cc:377] Logged in from keytab as kudu/127.0.197.126@KRBTEST.COM (short username kudu)
I20260929 16:44:39.179639 13142 webserver.cc:460] Webserver started at http://127.0.197.126:42967/ using document root <none> and password file <none>
I20260929 16:44:39.180356 13142 fs_manager.cc:359] Metadata directory not provided
I20260929 16:44:39.180465 13142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:44:39.180748 13142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:44:39 dist-test-slave-l31f krb5kdc[12698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700279, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for HTTP/127.0.197.84@KRBTEST.COM
I20260929 16:44:39.251281 13142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/master-0/data/instance:
uuid: "9a77f6403a744551837636e865e63ada"
format_stamp: "Formatted at 2026-09-29 16:44:39 on dist-test-slave-l31f"
server_key: "41208e65987be9620f1f215f6419b369"
server_key_iv: "f4640d819c476c125d1f9cffe36ddb79"
server_key_version: "kuduclusterkey@0"
I20260929 16:44:39.252564 13142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/master-0/wal/instance:
uuid: "9a77f6403a744551837636e865e63ada"
format_stamp: "Formatted at 2026-09-29 16:44:39 on dist-test-slave-l31f"
server_key: "41208e65987be9620f1f215f6419b369"
server_key_iv: "f4640d819c476c125d1f9cffe36ddb79"
server_key_version: "kuduclusterkey@0"
I20260929 16:44:39.260634 13142 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20260929 16:44:39.279318 13157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:44:39.281088 13142 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260929 16:44:39.281347 13142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "9a77f6403a744551837636e865e63ada"
format_stamp: "Formatted at 2026-09-29 16:44:39 on dist-test-slave-l31f"
server_key: "41208e65987be9620f1f215f6419b369"
server_key_iv: "f4640d819c476c125d1f9cffe36ddb79"
server_key_version: "kuduclusterkey@0"
I20260929 16:44:39.281605 13142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:44:39.298666 13142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:44:39.303310 13142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:44:39.303646 13142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:44:39.325908 13142 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.126:39101
I20260929 16:44:39.325908 13208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.126:39101 every 8 connection(s)
I20260929 16:44:39.327430 13142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:44:39.331476 13209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:44:39.336031 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 13142
I20260929 16:44:39.336256 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/master-0/wal/instance
I20260929 16:44:39.336932 789 external_mini_cluster.cc:1428] Setting key 6b0aa44fb251c34825350b754e339943
I20260929 16:44:39.343513 13209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a77f6403a744551837636e865e63ada: Bootstrap starting.
I20260929 16:44:39.346894 13209 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a77f6403a744551837636e865e63ada: Neither blocks nor log segments found. Creating new log.
I20260929 16:44:39.348110 13209 log.cc:824] T 00000000000000000000000000000000 P 9a77f6403a744551837636e865e63ada: Log is configured to *not* fsync() on all Append() calls
I20260929 16:44:39.352577 13209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a77f6403a744551837636e865e63ada: No bootstrap required, opened a new log
Sep 29 16:44:39 dist-test-slave-l31f krb5kdc[12698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:44:39.364305 13209 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a77f6403a744551837636e865e63ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a77f6403a744551837636e865e63ada" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 39101 } }
I20260929 16:44:39.364715 13212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:39.339125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58276 (local address 127.0.197.126:39101)
0929 16:44:39.340068 (+ 943us) server_negotiation.cc:206] Beginning negotiation
0929 16:44:39.340083 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:44:39.340135 (+ 52us) server_negotiation.cc:407] Connection header received
0929 16:44:39.342097 (+ 1962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:44:39.342126 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:44:39.342606 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:44:39.343181 (+ 575us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:44:39.344737 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:39.346229 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:39.347853 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:39.348925 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:39.349192 (+ 267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:39.355069 (+ 5877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:44:39.355088 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:44:39.355114 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:44:39.355180 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:44:39.359426 (+ 4246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:39.360308 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:39.360318 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:39.360329 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:39.360564 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:39.360931 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:39.360937 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:39.360940 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:39.361885 (+ 945us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:44:39.362236 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0929 16:44:39.362675 (+ 439us) server_negotiation.cc:299] Negotiation successful
0929 16:44:39.363231 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":421,"threads_started":1}
I20260929 16:44:39.364840 13209 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a77f6403a744551837636e865e63ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:44:39.364943 13209 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a77f6403a744551837636e865e63ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a77f6403a744551837636e865e63ada, State: Initialized, Role: FOLLOWER
I20260929 16:44:39.365739 13209 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a77f6403a744551837636e865e63ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a77f6403a744551837636e865e63ada" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 39101 } }
I20260929 16:44:39.366060 13209 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a77f6403a744551837636e865e63ada [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:44:39.366211 13209 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a77f6403a744551837636e865e63ada [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:44:39.366417 13209 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a77f6403a744551837636e865e63ada [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:44:39.369403 13209 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a77f6403a744551837636e865e63ada [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a77f6403a744551837636e865e63ada" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 39101 } }
I20260929 16:44:39.369979 13209 leader_election.cc:302] T 00000000000000000000000000000000 P 9a77f6403a744551837636e865e63ada [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a77f6403a744551837636e865e63ada; no voters:
I20260929 16:44:39.370782 13209 leader_election.cc:288] T 00000000000000000000000000000000 P 9a77f6403a744551837636e865e63ada [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:44:39.370904 13214 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a77f6403a744551837636e865e63ada [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:44:39.371624 13214 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a77f6403a744551837636e865e63ada [term 1 LEADER]: Becoming Leader. State: Replica: 9a77f6403a744551837636e865e63ada, State: Running, Role: LEADER
I20260929 16:44:39.372218 13214 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a77f6403a744551837636e865e63ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a77f6403a744551837636e865e63ada" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 39101 } }
I20260929 16:44:39.373242 13209 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a77f6403a744551837636e865e63ada [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:44:39.375554 13216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a77f6403a744551837636e865e63ada [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a77f6403a744551837636e865e63ada. Latest consensus state: current_term: 1 leader_uuid: "9a77f6403a744551837636e865e63ada" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a77f6403a744551837636e865e63ada" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 39101 } } }
I20260929 16:44:39.376113 13216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a77f6403a744551837636e865e63ada [sys.catalog]: This master's current role is: LEADER
I20260929 16:44:39.375620 13215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a77f6403a744551837636e865e63ada [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a77f6403a744551837636e865e63ada" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a77f6403a744551837636e865e63ada" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 39101 } } }
I20260929 16:44:39.376663 13215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a77f6403a744551837636e865e63ada [sys.catalog]: This master's current role is: LEADER
I20260929 16:44:39.384737 13220 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:44:39.391423 13220 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:44:39.392393 13209 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 16:44:39.410758 13220 catalog_manager.cc:1347] Generated new cluster ID: 8a983c3b89404e70b536feadcf0b38c5
I20260929 16:44:39.411104 13220 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:44:39.441139 13220 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:44:39.442690 13220 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:44:39.452857 13220 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a77f6403a744551837636e865e63ada: Generated new TSK 0
I20260929 16:44:39.453949 13220 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.197.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab.
Entry for principal kudu/127.0.197.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab.
I20260929 16:44:40.279594 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.197.84:59275/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.65:0
--local_ip_for_outbound_sockets=127.0.197.65
--webserver_interface=127.0.197.65
--webserver_port=0
--tserver_master_addrs=127.0.197.126:39101
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab
--principal=kudu/127.0.197.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:36181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:44:40 dist-test-slave-l31f krb5kdc[12698](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790700280, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 16:44:40.744596 13254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:44:40.745038 13254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:44:40.745255 13254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:44:40.757562 13254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:44:40.757800 13254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:44:40.758029 13254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.65
I20260929 16:44:40.773224 13254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:36181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.197.84:59275/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab
--principal=kudu/127.0.197.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.197.65
--webserver_port=0
--tserver_master_addrs=127.0.197.126:39101
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.65
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:44:40.774793 13254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:44:40.776507 13254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:44:40.801903 13267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:44:40.801957 13265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:44:40.802201 13264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:44:40.805441 13254 server_base.cc:1034] running on GCE node
I20260929 16:44:40.806318 13254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:44:40.807924 13254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:44:40.809211 13254 hybrid_clock.cc:648] HybridClock initialized: now 1790700280809132 us; error 66 us; skew 500 ppm
I20260929 16:44:40.809653 13254 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:44:40 dist-test-slave-l31f krb5kdc[12698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700280, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:44:40.813364 13254 init.cc:377] Logged in from keytab as kudu/127.0.197.65@KRBTEST.COM (short username kudu)
I20260929 16:44:40.815307 13254 webserver.cc:460] Webserver started at http://127.0.197.65:39589/ using document root <none> and password file <none>
I20260929 16:44:40.816114 13254 fs_manager.cc:359] Metadata directory not provided
I20260929 16:44:40.816252 13254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:44:40.816607 13254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:44:40 dist-test-slave-l31f krb5kdc[12698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700280, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.65@KRBTEST.COM for HTTP/127.0.197.84@KRBTEST.COM
I20260929 16:44:40.846527 13254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-0/data/instance:
uuid: "2e03bac8f6824f2ba8b3becabb46d105"
format_stamp: "Formatted at 2026-09-29 16:44:40 on dist-test-slave-l31f"
server_key: "a2d2138ace96735285b67b0e25b4fab5"
server_key_iv: "627ce2b40381328168417fa701a12812"
server_key_version: "kuduclusterkey@0"
I20260929 16:44:40.847848 13254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-0/wal/instance:
uuid: "2e03bac8f6824f2ba8b3becabb46d105"
format_stamp: "Formatted at 2026-09-29 16:44:40 on dist-test-slave-l31f"
server_key: "a2d2138ace96735285b67b0e25b4fab5"
server_key_iv: "627ce2b40381328168417fa701a12812"
server_key_version: "kuduclusterkey@0"
I20260929 16:44:40.858536 13254 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.007s sys 0.001s
I20260929 16:44:40.891897 13274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:44:40.894268 13254 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.011s sys 0.003s
I20260929 16:44:40.894593 13254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-0/wal
uuid: "2e03bac8f6824f2ba8b3becabb46d105"
format_stamp: "Formatted at 2026-09-29 16:44:40 on dist-test-slave-l31f"
server_key: "a2d2138ace96735285b67b0e25b4fab5"
server_key_iv: "627ce2b40381328168417fa701a12812"
server_key_version: "kuduclusterkey@0"
I20260929 16:44:40.894999 13254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:44:40.957254 13254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:44:40.963414 13254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:44:40.963876 13254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:44:40.971668 13254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:44:40.980242 13254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:44:40.980472 13254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 16:44:40.980705 13254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:44:40.980840 13254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 16:44:41 dist-test-slave-l31f krb5kdc[12698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700280, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for HTTP/127.0.197.84@KRBTEST.COM
I20260929 16:44:41.135442 13254 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.65:39989
I20260929 16:44:41.135480 13386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.65:39989 every 8 connection(s)
I20260929 16:44:41.137145 13254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
I20260929 16:44:41.142099 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 13254
I20260929 16:44:41.142329 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-0/wal/instance
I20260929 16:44:41.143117 789 external_mini_cluster.cc:1428] Setting key 88f839a0e4bc5978af9c51240f9ed09f
WARNING: no policy specified for kudu/127.0.197.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.66@KRBTEST.COM" created.
Sep 29 16:44:41 dist-test-slave-l31f krb5kdc[12698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700280, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.65@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:44:41.208271 13388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:41.146910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:39101 (local address 127.0.197.65:48051)
0929 16:44:41.178182 (+ 31272us) negotiation.cc:107] Waiting for socket to connect
0929 16:44:41.178243 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 16:44:41.180807 (+ 2564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:44:41.182432 (+ 1625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:44:41.182448 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:44:41.183226 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:44:41.184115 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:44:41.184143 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:44:41.185703 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:44:41.185709 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:44:41.187857 (+ 2148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:44:41.187876 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:44:41.189118 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:44:41.189126 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:44:41.189430 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:41.193410 (+ 3980us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:44:41.193446 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:44:41.197884 (+ 4438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:44:41.202629 (+ 4745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:44:41.202639 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:44:41.202661 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:44:41.203451 (+ 790us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:44:41.204004 (+ 553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:44:41.204010 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:44:41.204013 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:44:41.204342 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:44:41.205166 (+ 824us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:44:41.205175 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:44:41.205726 (+ 551us) client_negotiation.cc:769] Sending connection context
0929 16:44:41.206066 (+ 340us) client_negotiation.cc:240] Negotiation successful
0929 16:44:41.206705 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30926,"thread_start_us":34883,"threads_started":1}
I20260929 16:44:41.209579 13389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:41.147582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:48051 (local address 127.0.197.126:39101)
0929 16:44:41.177478 (+ 29896us) server_negotiation.cc:206] Beginning negotiation
0929 16:44:41.177486 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:44:41.181396 (+ 3910us) server_negotiation.cc:407] Connection header received
0929 16:44:41.181482 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:44:41.181491 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:44:41.181617 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:44:41.181738 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:44:41.184385 (+ 2647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:41.185463 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:41.188065 (+ 2602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:41.188683 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:41.188806 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:41.198425 (+ 9619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:44:41.198434 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:44:41.198438 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:44:41.198504 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:44:41.202338 (+ 3834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:41.203654 (+ 1316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:41.203662 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:41.203665 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:41.203847 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:41.204492 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:41.204498 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:41.204501 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:41.204926 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:44:41.205081 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 16:44:41.208925 (+ 3844us) server_negotiation.cc:299] Negotiation successful
0929 16:44:41.209186 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29741,"thread_start_us":29918,"threads_started":1}
I20260929 16:44:41.212096 13387 heartbeater.cc:347] Connected to a master server at 127.0.197.126:39101
I20260929 16:44:41.212491 13387 heartbeater.cc:464] Registering TS with master...
I20260929 16:44:41.213418 13387 heartbeater.cc:511] Master 127.0.197.126:39101 requested a full tablet report, sending...
I20260929 16:44:41.216144 13174 ts_manager.cc:194] Registered new tserver with Master: 2e03bac8f6824f2ba8b3becabb46d105 (127.0.197.65:39989)
I20260929 16:44:41.217726 13174 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.65@KRBTEST.COM'} at 127.0.197.65:48051
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab.
Entry for principal kudu/127.0.197.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab.
W20260929 16:44:41.250669 789 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.197.66: real 0.104s user 0.000s sys 0.018s
I20260929 16:44:41.251526 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.197.84:59275/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.66:0
--local_ip_for_outbound_sockets=127.0.197.66
--webserver_interface=127.0.197.66
--webserver_port=0
--tserver_master_addrs=127.0.197.126:39101
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab
--principal=kudu/127.0.197.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:36181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:44:41 dist-test-slave-l31f krb5kdc[12698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700280, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for HTTP/127.0.197.84@KRBTEST.COM
W20260929 16:44:41.696044 13393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:44:41.696334 13393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:44:41.696494 13393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:44:41.705477 13393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:44:41.705631 13393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:44:41.705806 13393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.66
I20260929 16:44:41.716894 13393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:36181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.197.84:59275/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab
--principal=kudu/127.0.197.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.197.66
--webserver_port=0
--tserver_master_addrs=127.0.197.126:39101
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.66
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:44:41.718289 13393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:44:41.719973 13393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:44:41.736238 13410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:44:41.736253 13408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:44:41.736327 13407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:44:41.737179 13393 server_base.cc:1034] running on GCE node
I20260929 16:44:41.737875 13393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:44:41.738996 13393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:44:41.740348 13393 hybrid_clock.cc:648] HybridClock initialized: now 1790700281740298 us; error 44 us; skew 500 ppm
I20260929 16:44:41.740775 13393 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:44:41 dist-test-slave-l31f krb5kdc[12698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:44:41.751502 13393 init.cc:377] Logged in from keytab as kudu/127.0.197.66@KRBTEST.COM (short username kudu)
I20260929 16:44:41.753304 13393 webserver.cc:460] Webserver started at http://127.0.197.66:33903/ using document root <none> and password file <none>
I20260929 16:44:41.754103 13393 fs_manager.cc:359] Metadata directory not provided
I20260929 16:44:41.754273 13393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:44:41.754616 13393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:44:41 dist-test-slave-l31f krb5kdc[12698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.66@KRBTEST.COM for HTTP/127.0.197.84@KRBTEST.COM
I20260929 16:44:41.772519 13393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-1/data/instance:
uuid: "acac5948133b4015a4209e1058426c67"
format_stamp: "Formatted at 2026-09-29 16:44:41 on dist-test-slave-l31f"
server_key: "ccddf0ff88e4656fdfb903424057e969"
server_key_iv: "47e205245f4700ab8e22abaca6e5df4a"
server_key_version: "kuduclusterkey@0"
I20260929 16:44:41.773607 13393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-1/wal/instance:
uuid: "acac5948133b4015a4209e1058426c67"
format_stamp: "Formatted at 2026-09-29 16:44:41 on dist-test-slave-l31f"
server_key: "ccddf0ff88e4656fdfb903424057e969"
server_key_iv: "47e205245f4700ab8e22abaca6e5df4a"
server_key_version: "kuduclusterkey@0"
I20260929 16:44:41.780078 13393 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 16:44:41.797435 13417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:44:41.799580 13393 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:44:41.799808 13393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-1/wal
uuid: "acac5948133b4015a4209e1058426c67"
format_stamp: "Formatted at 2026-09-29 16:44:41 on dist-test-slave-l31f"
server_key: "ccddf0ff88e4656fdfb903424057e969"
server_key_iv: "47e205245f4700ab8e22abaca6e5df4a"
server_key_version: "kuduclusterkey@0"
I20260929 16:44:41.800091 13393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:44:41.817750 13393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:44:41.822803 13393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:44:41.823467 13393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:44:41.824630 13393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:44:41.826167 13393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:44:41.826249 13393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:44:41.826395 13393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:44:41.826460 13393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:44:41.867538 13393 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.66:44971
I20260929 16:44:41.867574 13529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.66:44971 every 8 connection(s)
I20260929 16:44:41.869081 13393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
I20260929 16:44:41.873167 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 13393
I20260929 16:44:41.873386 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-1/wal/instance
I20260929 16:44:41.874070 789 external_mini_cluster.cc:1428] Setting key e6f7dad5a2ce4f45f59329686a7dc343
Sep 29 16:44:41 dist-test-slave-l31f krb5kdc[12698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.66@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:44:41.893374 13532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:41.871959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:37791 (local address 127.0.197.126:39101)
0929 16:44:41.872439 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0929 16:44:41.872444 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:44:41.873813 (+ 1369us) server_negotiation.cc:407] Connection header received
0929 16:44:41.875938 (+ 2125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:44:41.875942 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:44:41.876023 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:44:41.876116 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:44:41.877731 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:41.878981 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:41.880855 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:41.881631 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:41.881790 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:41.886104 (+ 4314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:44:41.886110 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:44:41.886114 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:44:41.886155 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:44:41.889260 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:41.890427 (+ 1167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:41.890433 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:41.890435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:41.890581 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:41.891156 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:41.891161 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:41.891164 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:41.891493 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:44:41.891629 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 16:44:41.892734 (+ 1105us) server_negotiation.cc:299] Negotiation successful
0929 16:44:41.892935 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":248,"threads_started":1}
I20260929 16:44:41.894822 13533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:41.872635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:39101 (local address 127.0.197.66:37791)
0929 16:44:41.873540 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0929 16:44:41.873588 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 16:44:41.875625 (+ 2037us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:44:41.876423 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:44:41.876433 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:44:41.876890 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:44:41.877522 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:44:41.877541 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:44:41.879185 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:44:41.879189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:44:41.880652 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:44:41.880675 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:44:41.881909 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:44:41.881915 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:44:41.882204 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:41.883165 (+ 961us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:44:41.883191 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:44:41.885776 (+ 2585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:44:41.889468 (+ 3692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:44:41.889482 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:44:41.889505 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:44:41.890272 (+ 767us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:44:41.890707 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:44:41.890715 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:44:41.890721 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:44:41.891035 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:44:41.891650 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:44:41.891662 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:44:41.892209 (+ 547us) client_negotiation.cc:769] Sending connection context
0929 16:44:41.892558 (+ 349us) client_negotiation.cc:240] Negotiation successful
0929 16:44:41.893199 (+ 641us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":377,"threads_started":1}
I20260929 16:44:41.897120 13530 heartbeater.cc:347] Connected to a master server at 127.0.197.126:39101
I20260929 16:44:41.897529 13530 heartbeater.cc:464] Registering TS with master...
I20260929 16:44:41.898375 13530 heartbeater.cc:511] Master 127.0.197.126:39101 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.197.67@KRBTEST.COM; defaulting to no policy
I20260929 16:44:41.900661 13174 ts_manager.cc:194] Registered new tserver with Master: acac5948133b4015a4209e1058426c67 (127.0.197.66:44971)
I20260929 16:44:41.901641 13174 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.66@KRBTEST.COM'} at 127.0.197.66:37791
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab.
Entry for principal kudu/127.0.197.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab.
I20260929 16:44:41.934428 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.197.84:59275/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.67:0
--local_ip_for_outbound_sockets=127.0.197.67
--webserver_interface=127.0.197.67
--webserver_port=0
--tserver_master_addrs=127.0.197.126:39101
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab
--principal=kudu/127.0.197.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:36181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:44:42.222112 13387 heartbeater.cc:503] Master 127.0.197.126:39101 was elected leader, sending a full tablet report...
W20260929 16:44:42.284319 13536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:44:42.284641 13536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:44:42.284794 13536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:44:42.292997 13536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:44:42.293130 13536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:44:42.293319 13536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.67
I20260929 16:44:42.304255 13536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:36181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.197.84:59275/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab
--principal=kudu/127.0.197.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.197.67
--webserver_port=0
--tserver_master_addrs=127.0.197.126:39101
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.67
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:44:42.305693 13536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:44:42.307209 13536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:44:42.321133 13541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:44:42.321230 13542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:44:42.321206 13544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:44:42.322063 13536 server_base.cc:1034] running on GCE node
I20260929 16:44:42.322824 13536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:44:42.324146 13536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:44:42.325379 13536 hybrid_clock.cc:648] HybridClock initialized: now 1790700282325326 us; error 50 us; skew 500 ppm
I20260929 16:44:42.325750 13536 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:44:42 dist-test-slave-l31f krb5kdc[12698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700282, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:44:42.329371 13536 init.cc:377] Logged in from keytab as kudu/127.0.197.67@KRBTEST.COM (short username kudu)
I20260929 16:44:42.331288 13536 webserver.cc:460] Webserver started at http://127.0.197.67:41989/ using document root <none> and password file <none>
I20260929 16:44:42.332149 13536 fs_manager.cc:359] Metadata directory not provided
I20260929 16:44:42.332311 13536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:44:42.332649 13536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:44:42 dist-test-slave-l31f krb5kdc[12698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700282, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.67@KRBTEST.COM for HTTP/127.0.197.84@KRBTEST.COM
I20260929 16:44:42.348820 13536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-2/data/instance:
uuid: "ccb1dba7c1d344698180c2109a98b49c"
format_stamp: "Formatted at 2026-09-29 16:44:42 on dist-test-slave-l31f"
server_key: "4a4c7ec295b6a10f64fe57f27d92174b"
server_key_iv: "3b9c1c823a9ddf8f6de8cf07c6b6e682"
server_key_version: "kuduclusterkey@0"
I20260929 16:44:42.349768 13536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-2/wal/instance:
uuid: "ccb1dba7c1d344698180c2109a98b49c"
format_stamp: "Formatted at 2026-09-29 16:44:42 on dist-test-slave-l31f"
server_key: "4a4c7ec295b6a10f64fe57f27d92174b"
server_key_iv: "3b9c1c823a9ddf8f6de8cf07c6b6e682"
server_key_version: "kuduclusterkey@0"
I20260929 16:44:42.355902 13536 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 16:44:42.373371 13551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:44:42.375522 13536 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 16:44:42.375773 13536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-2/wal
uuid: "ccb1dba7c1d344698180c2109a98b49c"
format_stamp: "Formatted at 2026-09-29 16:44:42 on dist-test-slave-l31f"
server_key: "4a4c7ec295b6a10f64fe57f27d92174b"
server_key_iv: "3b9c1c823a9ddf8f6de8cf07c6b6e682"
server_key_version: "kuduclusterkey@0"
I20260929 16:44:42.376029 13536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:44:42.420090 13536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:44:42.424618 13536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:44:42.424958 13536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:44:42.426031 13536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:44:42.427624 13536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:44:42.427737 13536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:44:42.427896 13536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:44:42.427975 13536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:44:42.462996 13536 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.67:34281
I20260929 16:44:42.463068 13663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.67:34281 every 8 connection(s)
I20260929 16:44:42.464459 13536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
I20260929 16:44:42.470631 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 13536
I20260929 16:44:42.470839 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-2/wal/instance
I20260929 16:44:42.471469 789 external_mini_cluster.cc:1428] Setting key 606654e8bf9c8b254ed47dd857b83d61
Sep 29 16:44:42 dist-test-slave-l31f krb5kdc[12698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700282, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.67@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:44:42.486991 13666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:42.467364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:41405 (local address 127.0.197.126:39101)
0929 16:44:42.467881 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0929 16:44:42.467886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:44:42.468903 (+ 1017us) server_negotiation.cc:407] Connection header received
0929 16:44:42.470771 (+ 1868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:44:42.470777 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:44:42.470879 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:44:42.471050 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:44:42.473055 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:42.473957 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:42.475782 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:42.476366 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:42.476510 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:42.480247 (+ 3737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:44:42.480253 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:44:42.480257 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:44:42.480294 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:44:42.482980 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:42.483938 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:42.483943 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:42.483946 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:42.484115 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:42.484693 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:42.484698 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:42.484700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:42.485016 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:44:42.485158 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 16:44:42.486246 (+ 1088us) server_negotiation.cc:299] Negotiation successful
0929 16:44:42.486549 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":252,"threads_started":1}
I20260929 16:44:42.488301 13667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:42.467971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:39101 (local address 127.0.197.67:41405)
0929 16:44:42.468714 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0929 16:44:42.468759 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 16:44:42.470453 (+ 1694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:44:42.471365 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:44:42.471379 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:44:42.472002 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:44:42.472836 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:44:42.472868 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:44:42.474152 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:44:42.474162 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:44:42.475595 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:44:42.475634 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:44:42.476636 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:44:42.476647 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:44:42.476914 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:42.477709 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:44:42.477734 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:44:42.479943 (+ 2209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:44:42.483224 (+ 3281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:44:42.483233 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:44:42.483249 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:44:42.483781 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:44:42.484261 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:44:42.484265 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:44:42.484268 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:44:42.484496 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:44:42.485253 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:44:42.485262 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:44:42.485773 (+ 511us) client_negotiation.cc:769] Sending connection context
0929 16:44:42.486134 (+ 361us) client_negotiation.cc:240] Negotiation successful
0929 16:44:42.486748 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":299,"threads_started":1}
I20260929 16:44:42.489810 13664 heartbeater.cc:347] Connected to a master server at 127.0.197.126:39101
I20260929 16:44:42.490173 13664 heartbeater.cc:464] Registering TS with master...
I20260929 16:44:42.490958 13664 heartbeater.cc:511] Master 127.0.197.126:39101 requested a full tablet report, sending...
I20260929 16:44:42.492540 13174 ts_manager.cc:194] Registered new tserver with Master: ccb1dba7c1d344698180c2109a98b49c (127.0.197.67:34281)
I20260929 16:44:42.493366 13174 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.67@KRBTEST.COM'} at 127.0.197.67:41405
I20260929 16:44:42.500880 789 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:44:42.502730 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 13254
Sep 29 16:44:42 dist-test-slave-l31f krb5kdc[12698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.65@KRBTEST.COM
I20260929 16:44:42.521782 13669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:42.504003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47074 (local address 127.0.197.65:39989)
0929 16:44:42.504681 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0929 16:44:42.504690 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:44:42.504724 (+ 34us) server_negotiation.cc:407] Connection header received
0929 16:44:42.504879 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:44:42.504886 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:44:42.505203 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:44:42.505390 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:44:42.506882 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:42.508023 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:42.509728 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:42.510305 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:42.510463 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:42.515576 (+ 5113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:44:42.515592 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:44:42.515607 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:44:42.515655 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:44:42.518733 (+ 3078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:42.519694 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:42.519702 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:42.519704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:42.519889 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:42.520244 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:42.520247 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:42.520250 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:42.520625 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:44:42.520865 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0929 16:44:42.521127 (+ 262us) server_negotiation.cc:299] Negotiation successful
0929 16:44:42.521350 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":291,"threads_started":1}
I20260929 16:44:42.523342 13382 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:44:42.609763 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 13254
I20260929 16:44:42.627566 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 13393
Sep 29 16:44:42 dist-test-slave-l31f krb5kdc[12698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.66@KRBTEST.COM
I20260929 16:44:42.653198 13671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:42.628938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37744 (local address 127.0.197.66:44971)
0929 16:44:42.629758 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0929 16:44:42.629768 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:44:42.629812 (+ 44us) server_negotiation.cc:407] Connection header received
0929 16:44:42.629911 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:44:42.629919 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:44:42.630389 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:44:42.630566 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:44:42.632205 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:42.633415 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:42.635436 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:42.636118 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:42.636251 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:42.645588 (+ 9337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:44:42.645607 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:44:42.645624 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:44:42.645696 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:44:42.649019 (+ 3323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:42.649916 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:42.649924 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:42.649927 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:42.650082 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:42.650911 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:42.650918 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:42.650922 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:42.651719 (+ 797us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:44:42.652070 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0929 16:44:42.652559 (+ 489us) server_negotiation.cc:299] Negotiation successful
0929 16:44:42.652842 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":391,"threads_started":1}
I20260929 16:44:42.654901 13525 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:44:42.739059 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 13393
I20260929 16:44:42.755594 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 13536
Sep 29 16:44:42 dist-test-slave-l31f krb5kdc[12698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.67@KRBTEST.COM
I20260929 16:44:42.772846 13673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:42.756578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59450 (local address 127.0.197.67:34281)
0929 16:44:42.757195 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0929 16:44:42.757202 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:44:42.757234 (+ 32us) server_negotiation.cc:407] Connection header received
0929 16:44:42.757293 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:44:42.757299 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:44:42.757520 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:44:42.757703 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:44:42.759260 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:42.760287 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:42.761647 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:42.762245 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:42.762396 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:42.767183 (+ 4787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:44:42.767198 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:44:42.767215 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:44:42.767261 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:44:42.769918 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:42.770758 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:42.770766 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:42.770768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:42.770909 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:42.771392 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:42.771395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:42.771397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:42.771745 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:44:42.771983 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0929 16:44:42.772236 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 16:44:42.772437 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":324,"threads_started":1}
I20260929 16:44:42.774289 13659 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:44:42.859362 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 13536
I20260929 16:44:42.875574 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 13142
I20260929 16:44:42.876894 13204 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:44:42.986274 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 13142
I20260929 16:44:43.014055 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:39101
--webserver_interface=127.0.197.126
--webserver_port=42967
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:36181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.126:39101
--ranger_config_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.197.84:59275/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.197.84:59275/kms with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:44:43.366791 13676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:44:43.367237 13676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:44:43.367370 13676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:44:43.375267 13676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:44:43.375396 13676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:44:43.375523 13676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:44:43.375613 13676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:44:43.375690 13676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:44:43.386307 13676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:36181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.197.84:59275/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.197.126:39101
--ranger_config_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:39101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.197.126
--webserver_port=42967
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:44:43.387907 13676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:44:43.389256 13676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:44:43.401613 13681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:44:43.401654 13682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:44:43.401686 13684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:44:43.402429 13676 server_base.cc:1034] running on GCE node
I20260929 16:44:43.403364 13676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:44:43.405064 13676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:44:43.406351 13676 hybrid_clock.cc:648] HybridClock initialized: now 1790700283406288 us; error 59 us; skew 500 ppm
I20260929 16:44:43.406766 13676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:44:43 dist-test-slave-l31f krb5kdc[12698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700283, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:44:43.410187 13676 init.cc:377] Logged in from keytab as kudu/127.0.197.126@KRBTEST.COM (short username kudu)
I20260929 16:44:43.411937 13676 webserver.cc:460] Webserver started at http://127.0.197.126:42967/ using document root <none> and password file <none>
I20260929 16:44:43.412640 13676 fs_manager.cc:359] Metadata directory not provided
I20260929 16:44:43.412783 13676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 16:44:43 dist-test-slave-l31f krb5kdc[12698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700283, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for HTTP/127.0.197.84@KRBTEST.COM
I20260929 16:44:43.489408 13676 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:44:43.492677 13692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:44:43.494172 13676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:44:43.494378 13676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "9a77f6403a744551837636e865e63ada"
format_stamp: "Formatted at 2026-09-29 16:44:39 on dist-test-slave-l31f"
server_key: "41208e65987be9620f1f215f6419b369"
server_key_iv: "f4640d819c476c125d1f9cffe36ddb79"
server_key_version: "kuduclusterkey@0"
I20260929 16:44:43.495441 13676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:44:43.513660 13676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:44:43.518057 13676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:44:43.518404 13676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:44:43.537796 13676 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.126:39101
I20260929 16:44:43.537832 13743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.126:39101 every 8 connection(s)
I20260929 16:44:43.539482 13676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:44:43.540186 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 13676
I20260929 16:44:43.541213 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.197.84:59275/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.65:39989
--local_ip_for_outbound_sockets=127.0.197.65
--tserver_master_addrs=127.0.197.126:39101
--webserver_port=39589
--webserver_interface=127.0.197.65
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab
--principal=kudu/127.0.197.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:36181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.197.84:59275/kms with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:44:43.546964 13744 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:44:43.552798 13744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a77f6403a744551837636e865e63ada: Bootstrap starting.
I20260929 16:44:43.582939 13744 log.cc:824] T 00000000000000000000000000000000 P 9a77f6403a744551837636e865e63ada: Log is configured to *not* fsync() on all Append() calls
I20260929 16:44:43.594705 13744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a77f6403a744551837636e865e63ada: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:44:43.595566 13744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a77f6403a744551837636e865e63ada: Bootstrap complete.
I20260929 16:44:43.611440 13744 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a77f6403a744551837636e865e63ada [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a77f6403a744551837636e865e63ada" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 39101 } }
I20260929 16:44:43.613214 13744 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a77f6403a744551837636e865e63ada [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a77f6403a744551837636e865e63ada, State: Initialized, Role: FOLLOWER
I20260929 16:44:43.614252 13744 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a77f6403a744551837636e865e63ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a77f6403a744551837636e865e63ada" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 39101 } }
I20260929 16:44:43.614632 13744 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a77f6403a744551837636e865e63ada [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:44:43.614791 13744 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a77f6403a744551837636e865e63ada [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:44:43.615070 13744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a77f6403a744551837636e865e63ada [term 1 FOLLOWER]: Advancing to term 2
I20260929 16:44:43.620987 13744 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a77f6403a744551837636e865e63ada [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a77f6403a744551837636e865e63ada" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 39101 } }
I20260929 16:44:43.621647 13744 leader_election.cc:302] T 00000000000000000000000000000000 P 9a77f6403a744551837636e865e63ada [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a77f6403a744551837636e865e63ada; no voters:
I20260929 16:44:43.622284 13744 leader_election.cc:288] T 00000000000000000000000000000000 P 9a77f6403a744551837636e865e63ada [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 16:44:43.622452 13748 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a77f6403a744551837636e865e63ada [term 2 FOLLOWER]: Leader election won for term 2
I20260929 16:44:43.623838 13748 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a77f6403a744551837636e865e63ada [term 2 LEADER]: Becoming Leader. State: Replica: 9a77f6403a744551837636e865e63ada, State: Running, Role: LEADER
I20260929 16:44:43.624424 13744 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a77f6403a744551837636e865e63ada [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:44:43.624466 13748 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a77f6403a744551837636e865e63ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a77f6403a744551837636e865e63ada" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 39101 } }
I20260929 16:44:43.627060 13750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a77f6403a744551837636e865e63ada [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a77f6403a744551837636e865e63ada. Latest consensus state: current_term: 2 leader_uuid: "9a77f6403a744551837636e865e63ada" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a77f6403a744551837636e865e63ada" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 39101 } } }
I20260929 16:44:43.627468 13750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a77f6403a744551837636e865e63ada [sys.catalog]: This master's current role is: LEADER
I20260929 16:44:43.627082 13749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a77f6403a744551837636e865e63ada [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9a77f6403a744551837636e865e63ada" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a77f6403a744551837636e865e63ada" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 39101 } } }
I20260929 16:44:43.628191 13749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a77f6403a744551837636e865e63ada [sys.catalog]: This master's current role is: LEADER
I20260929 16:44:43.632122 13755 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:44:43.638381 13755 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:44:43.639297 13744 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 16:44:43.640151 13755 catalog_manager.cc:1259] Loaded cluster ID: 8a983c3b89404e70b536feadcf0b38c5
I20260929 16:44:43.640341 13755 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:44:43.645979 13755 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:44:43.648504 13755 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9a77f6403a744551837636e865e63ada: Loaded TSK: 0
I20260929 16:44:43.650907 13755 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 16:44:44.044656 13746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:44:44.045146 13746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:44:44.045328 13746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:44:44.057679 13746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:44:44.057858 13746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:44:44.058102 13746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.65
I20260929 16:44:44.074934 13746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:36181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.197.84:59275/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab
--principal=kudu/127.0.197.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.65:39989
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.197.65
--webserver_port=39589
--tserver_master_addrs=127.0.197.126:39101
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.65
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:44:44.077030 13746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:44:44.079098 13746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:44:44.130152 13782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:44:44.137876 13781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:44:44.138437 13784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:44:44.139571 13746 server_base.cc:1034] running on GCE node
I20260929 16:44:44.140527 13746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:44:44.144883 13746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:44:44.146541 13746 hybrid_clock.cc:648] HybridClock initialized: now 1790700284146452 us; error 83 us; skew 500 ppm
I20260929 16:44:44.147184 13746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:44:44 dist-test-slave-l31f krb5kdc[12698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700284, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:44:44.152591 13746 init.cc:377] Logged in from keytab as kudu/127.0.197.65@KRBTEST.COM (short username kudu)
I20260929 16:44:44.173868 13746 webserver.cc:460] Webserver started at http://127.0.197.65:39589/ using document root <none> and password file <none>
I20260929 16:44:44.175493 13746 fs_manager.cc:359] Metadata directory not provided
I20260929 16:44:44.175840 13746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 16:44:44 dist-test-slave-l31f krb5kdc[12698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700284, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.65@KRBTEST.COM for HTTP/127.0.197.84@KRBTEST.COM
I20260929 16:44:44.209105 13746 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.005s sys 0.000s
I20260929 16:44:44.223901 13793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:44:44.236945 13746 fs_manager.cc:725] Time spent opening block manager: real 0.026s user 0.005s sys 0.008s
I20260929 16:44:44.237357 13746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-0/wal
uuid: "2e03bac8f6824f2ba8b3becabb46d105"
format_stamp: "Formatted at 2026-09-29 16:44:40 on dist-test-slave-l31f"
server_key: "a2d2138ace96735285b67b0e25b4fab5"
server_key_iv: "627ce2b40381328168417fa701a12812"
server_key_version: "kuduclusterkey@0"
I20260929 16:44:44.239266 13746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:44:44.289268 13746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:44:44.295867 13746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:44:44.296350 13746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:44:44.297932 13746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:44:44.300040 13746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:44:44.300185 13746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:44:44.300329 13746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:44:44.300439 13746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:44:44.378998 13746 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.65:39989
I20260929 16:44:44.379123 13905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.65:39989 every 8 connection(s)
I20260929 16:44:44.380705 13746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
I20260929 16:44:44.387327 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 13746
I20260929 16:44:44.388181 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.197.84:59275/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.66:44971
--local_ip_for_outbound_sockets=127.0.197.66
--tserver_master_addrs=127.0.197.126:39101
--webserver_port=33903
--webserver_interface=127.0.197.66
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab
--principal=kudu/127.0.197.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:36181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.197.84:59275/kms with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:44:44 dist-test-slave-l31f krb5kdc[12698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700284, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.65@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:44:44.433763 13908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:44.384577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:36323 (local address 127.0.197.126:39101)
0929 16:44:44.400864 (+ 16287us) server_negotiation.cc:206] Beginning negotiation
0929 16:44:44.400881 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 16:44:44.400938 (+ 57us) server_negotiation.cc:407] Connection header received
0929 16:44:44.402781 (+ 1843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:44:44.402802 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:44:44.403532 (+ 730us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:44:44.404075 (+ 543us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:44:44.407587 (+ 3512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:44.410027 (+ 2440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:44.413508 (+ 3481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:44.414929 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:44.415254 (+ 325us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:44.421220 (+ 5966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:44:44.421237 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:44:44.421266 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:44:44.421347 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:44:44.426747 (+ 5400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:44.428326 (+ 1579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:44.428341 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:44.428355 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:44.428653 (+ 298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:44.429405 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:44.429415 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:44.429422 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:44.430376 (+ 954us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:44:44.430665 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0929 16:44:44.431843 (+ 1178us) server_negotiation.cc:299] Negotiation successful
0929 16:44:44.432329 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16001,"thread_start_us":15754,"threads_started":1}
I20260929 16:44:44.433760 13909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:44.384683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:39101 (local address 127.0.197.65:36323)
0929 16:44:44.396384 (+ 11701us) negotiation.cc:107] Waiting for socket to connect
0929 16:44:44.396479 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0929 16:44:44.401754 (+ 5275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:44:44.405295 (+ 3541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:44:44.405315 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:44:44.406296 (+ 981us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:44:44.407178 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:44:44.407224 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:44:44.410387 (+ 3163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:44:44.410398 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:44:44.413170 (+ 2772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:44:44.413202 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:44:44.415764 (+ 2562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:44:44.415775 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:44:44.416180 (+ 405us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:44.417157 (+ 977us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:44:44.417200 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:44:44.420709 (+ 3509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:44:44.427106 (+ 6397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:44:44.427121 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:44:44.427147 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:44:44.428006 (+ 859us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:44:44.428854 (+ 848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:44:44.428862 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:44:44.428869 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:44:44.429213 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:44:44.430659 (+ 1446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:44:44.430669 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:44:44.431173 (+ 504us) client_negotiation.cc:769] Sending connection context
0929 16:44:44.431546 (+ 373us) client_negotiation.cc:240] Negotiation successful
0929 16:44:44.432261 (+ 715us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11126,"thread_start_us":14286,"threads_started":1}
I20260929 16:44:44.438814 13906 heartbeater.cc:347] Connected to a master server at 127.0.197.126:39101
I20260929 16:44:44.439263 13906 heartbeater.cc:464] Registering TS with master...
I20260929 16:44:44.440338 13906 heartbeater.cc:511] Master 127.0.197.126:39101 requested a full tablet report, sending...
W20260929 16:44:44.444509 13906 heartbeater.cc:644] Failed to heartbeat to 127.0.197.126:39101 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 16:44:44.444797 13906 heartbeater.cc:464] Registering TS with master...
I20260929 16:44:44.445153 13906 heartbeater.cc:511] Master 127.0.197.126:39101 requested a full tablet report, sending...
I20260929 16:44:44.446981 13906 heartbeater.cc:464] Registering TS with master...
I20260929 16:44:44.447481 13906 heartbeater.cc:511] Master 127.0.197.126:39101 requested a full tablet report, sending...
W20260929 16:44:44.448913 13906 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 16:44:44.912062 13910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:44:44.912544 13910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:44:44.912734 13910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:44:44.925307 13910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:44:44.925504 13910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:44:44.925719 13910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.66
I20260929 16:44:44.942549 13910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:36181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.197.84:59275/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab
--principal=kudu/127.0.197.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.66:44971
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.197.66
--webserver_port=33903
--tserver_master_addrs=127.0.197.126:39101
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.66
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:44:44.944804 13910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:44:44.947050 13910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:44:44.985006 13925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:44:44.985095 13924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:44:44.986348 13910 server_base.cc:1034] running on GCE node
W20260929 16:44:44.986377 13928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:44:44.987733 13910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:44:44.995667 13910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:44:45.002039 13910 hybrid_clock.cc:648] HybridClock initialized: now 1790700285001886 us; error 53 us; skew 500 ppm
I20260929 16:44:45.002975 13910 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:44:45 dist-test-slave-l31f krb5kdc[12698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700285, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:44:45.008916 13910 init.cc:377] Logged in from keytab as kudu/127.0.197.66@KRBTEST.COM (short username kudu)
I20260929 16:44:45.023679 13910 webserver.cc:460] Webserver started at http://127.0.197.66:33903/ using document root <none> and password file <none>
I20260929 16:44:45.024605 13910 fs_manager.cc:359] Metadata directory not provided
I20260929 16:44:45.024844 13910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 16:44:45 dist-test-slave-l31f krb5kdc[12698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700285, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.66@KRBTEST.COM for HTTP/127.0.197.84@KRBTEST.COM
I20260929 16:44:45.057768 13910 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 16:44:45.061352 13937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:44:45.063282 13910 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:44:45.063496 13910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-1/wal
uuid: "acac5948133b4015a4209e1058426c67"
format_stamp: "Formatted at 2026-09-29 16:44:41 on dist-test-slave-l31f"
server_key: "ccddf0ff88e4656fdfb903424057e969"
server_key_iv: "47e205245f4700ab8e22abaca6e5df4a"
server_key_version: "kuduclusterkey@0"
I20260929 16:44:45.064671 13910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:44:45.079463 13910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
Sep 29 16:44:45 dist-test-slave-l31f krb5kdc[12698](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790700285, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:44:45.084205 13910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:44:45.084508 13910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:44:45.085625 13910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:44:45.087216 13910 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:44:45.087328 13910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:44:45.087463 13910 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:44:45.087546 13910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:44:45.155980 13910 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.66:44971
I20260929 16:44:45.156023 14049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.66:44971 every 8 connection(s)
I20260929 16:44:45.157541 13910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
I20260929 16:44:45.161733 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 13910
I20260929 16:44:45.162459 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.197.84:59275/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.67:34281
--local_ip_for_outbound_sockets=127.0.197.67
--tserver_master_addrs=127.0.197.126:39101
--webserver_port=41989
--webserver_interface=127.0.197.67
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab
--principal=kudu/127.0.197.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:36181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.197.84:59275/kms with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:44:45 dist-test-slave-l31f krb5kdc[12698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700285, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.66@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:44:45.184119 14052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:45.160619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:41925 (local address 127.0.197.126:39101)
0929 16:44:45.161148 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0929 16:44:45.161155 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:44:45.163107 (+ 1952us) server_negotiation.cc:407] Connection header received
0929 16:44:45.164438 (+ 1331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:44:45.164446 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:44:45.164540 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:44:45.164669 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:44:45.166237 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:45.167064 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:45.168749 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:45.169246 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:45.169421 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:45.175328 (+ 5907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:44:45.175339 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:44:45.175348 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:44:45.175402 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:44:45.179543 (+ 4141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:45.180737 (+ 1194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:45.180744 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:45.180747 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:45.180918 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:45.181543 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:45.181550 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:45.181554 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:45.181993 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:44:45.182132 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 16:44:45.183483 (+ 1351us) server_negotiation.cc:299] Negotiation successful
0929 16:44:45.183760 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":253,"threads_started":1}
I20260929 16:44:45.186035 14053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:45.161206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:39101 (local address 127.0.197.66:41925)
0929 16:44:45.162030 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0929 16:44:45.162089 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 16:44:45.164047 (+ 1958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:44:45.164955 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:44:45.164965 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:44:45.165414 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:44:45.166027 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:44:45.166045 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:44:45.167237 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:44:45.167243 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:44:45.168564 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:44:45.168579 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:44:45.169534 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:44:45.169543 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:44:45.169924 (+ 381us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:45.170708 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:44:45.170749 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:44:45.174914 (+ 4165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:44:45.179828 (+ 4914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:44:45.179839 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:44:45.179861 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:44:45.180542 (+ 681us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:44:45.181089 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:44:45.181095 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:44:45.181100 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:44:45.181385 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:44:45.182363 (+ 978us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:44:45.182373 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:44:45.182917 (+ 544us) client_negotiation.cc:769] Sending connection context
0929 16:44:45.184403 (+ 1486us) client_negotiation.cc:240] Negotiation successful
0929 16:44:45.184952 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":340,"threads_started":1}
I20260929 16:44:45.188266 14050 heartbeater.cc:347] Connected to a master server at 127.0.197.126:39101
I20260929 16:44:45.188652 14050 heartbeater.cc:464] Registering TS with master...
I20260929 16:44:45.189674 14050 heartbeater.cc:511] Master 127.0.197.126:39101 requested a full tablet report, sending...
I20260929 16:44:45.192284 13709 ts_manager.cc:194] Registered new tserver with Master: acac5948133b4015a4209e1058426c67 (127.0.197.66:44971)
I20260929 16:44:45.194710 13709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.66@KRBTEST.COM'} at 127.0.197.66:41925
I20260929 16:44:45.449133 13906 heartbeater.cc:464] Registering TS with master...
I20260929 16:44:45.449646 13906 heartbeater.cc:511] Master 127.0.197.126:39101 requested a full tablet report, sending...
I20260929 16:44:45.451241 13709 ts_manager.cc:194] Registered new tserver with Master: 2e03bac8f6824f2ba8b3becabb46d105 (127.0.197.65:39989)
I20260929 16:44:45.452425 13709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.65@KRBTEST.COM'} at 127.0.197.65:36323
Sep 29 16:44:45 dist-test-slave-l31f krb5kdc[12698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700285, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for HTTP/127.0.197.84@KRBTEST.COM
W20260929 16:44:45.699659 14054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:44:45.700139 14054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:44:45.700333 14054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:44:45.708609 14054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:44:45.708783 14054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:44:45.708958 14054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.67
I20260929 16:44:45.719605 14054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:36181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.197.84:59275/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab
--principal=kudu/127.0.197.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.67:34281
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.197.67
--webserver_port=41989
--tserver_master_addrs=127.0.197.126:39101
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.67
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:44:45.721093 14054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:44:45.722447 14054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:44:45.740993 14060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:44:45.742377 14061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:44:45.753085 14063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:44:45.753376 14054 server_base.cc:1034] running on GCE node
I20260929 16:44:45.754453 14054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:44:45.759152 14054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:44:45.760593 14054 hybrid_clock.cc:648] HybridClock initialized: now 1790700285760507 us; error 81 us; skew 500 ppm
I20260929 16:44:45.761042 14054 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:44:45 dist-test-slave-l31f krb5kdc[12698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700285, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:44:45.765478 14054 init.cc:377] Logged in from keytab as kudu/127.0.197.67@KRBTEST.COM (short username kudu)
I20260929 16:44:45.768684 14054 webserver.cc:460] Webserver started at http://127.0.197.67:41989/ using document root <none> and password file <none>
I20260929 16:44:45.769784 14054 fs_manager.cc:359] Metadata directory not provided
I20260929 16:44:45.769980 14054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 16:44:45 dist-test-slave-l31f krb5kdc[12698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700285, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.67@KRBTEST.COM for HTTP/127.0.197.84@KRBTEST.COM
I20260929 16:44:45.798619 14054 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 16:44:45.802640 14071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:44:45.804752 14054 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260929 16:44:45.805049 14054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-2/wal
uuid: "ccb1dba7c1d344698180c2109a98b49c"
format_stamp: "Formatted at 2026-09-29 16:44:42 on dist-test-slave-l31f"
server_key: "4a4c7ec295b6a10f64fe57f27d92174b"
server_key_iv: "3b9c1c823a9ddf8f6de8cf07c6b6e682"
server_key_version: "kuduclusterkey@0"
I20260929 16:44:45.806563 14054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:44:45.833128 14054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:44:45.839303 14054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:44:45.839695 14054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:44:45.840976 14054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:44:45.843163 14054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:44:45.843353 14054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:44:45.843505 14054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:44:45.843613 14054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:44:45.918576 14054 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.67:34281
I20260929 16:44:45.918655 14183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.67:34281 every 8 connection(s)
I20260929 16:44:45.920802 14054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
I20260929 16:44:45.928382 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 14054
Sep 29 16:44:45 dist-test-slave-l31f krb5kdc[12698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700285, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.67@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:44:45.976259 14186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:45.929512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43144 (local address 127.0.197.126:39101)
0929 16:44:45.933133 (+ 3621us) server_negotiation.cc:206] Beginning negotiation
0929 16:44:45.933141 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:44:45.940042 (+ 6901us) server_negotiation.cc:407] Connection header received
0929 16:44:45.947881 (+ 7839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:44:45.947893 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:44:45.948055 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:44:45.948233 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:44:45.950566 (+ 2333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:45.951965 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:45.955536 (+ 3571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:45.956210 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:45.956356 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:45.962206 (+ 5850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:44:45.962214 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:44:45.962218 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:44:45.962283 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:44:45.968964 (+ 6681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:45.971253 (+ 2289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:45.971260 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:45.971265 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:45.971490 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:45.973805 (+ 2315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:45.973812 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:45.973816 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:45.974278 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:44:45.974617 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0929 16:44:45.975208 (+ 591us) server_negotiation.cc:299] Negotiation successful
0929 16:44:45.975514 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3464,"thread_start_us":3351,"threads_started":1}
Sep 29 16:44:45 dist-test-slave-l31f krb5kdc[12698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700285, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for HTTP/127.0.197.84@KRBTEST.COM
I20260929 16:44:45.984790 14189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:45.942051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:39101 (local address 127.0.197.67:52641)
0929 16:44:45.951542 (+ 9491us) negotiation.cc:107] Waiting for socket to connect
0929 16:44:45.951606 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 16:44:45.956607 (+ 5001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:44:45.958262 (+ 1655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:44:45.958276 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:44:45.959090 (+ 814us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:44:45.960123 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:44:45.960160 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:44:45.963220 (+ 3060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:44:45.963229 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:44:45.965660 (+ 2431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:44:45.965685 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:44:45.969379 (+ 3694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:44:45.969388 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:44:45.969752 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:45.971426 (+ 1674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:44:45.971474 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:44:45.974385 (+ 2911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:44:45.979555 (+ 5170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:44:45.979566 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:44:45.979587 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:44:45.980260 (+ 673us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:44:45.980718 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:44:45.980722 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:44:45.980725 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:44:45.980976 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:44:45.981661 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:44:45.981670 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:44:45.982363 (+ 693us) client_negotiation.cc:769] Sending connection context
0929 16:44:45.982687 (+ 324us) client_negotiation.cc:240] Negotiation successful
0929 16:44:45.983362 (+ 675us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9089,"thread_start_us":13202,"threads_started":1}
I20260929 16:44:45.986016 14190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:45.943066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:52641 (local address 127.0.197.126:39101)
0929 16:44:45.957623 (+ 14557us) server_negotiation.cc:206] Beginning negotiation
0929 16:44:45.957628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:44:45.957644 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:44:45.957694 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:44:45.957698 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:44:45.957770 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:44:45.957921 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:44:45.960454 (+ 2533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:45.961948 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:45.966324 (+ 4376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:45.966973 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:45.967154 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:45.975765 (+ 8611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:44:45.975772 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:44:45.975776 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:44:45.975819 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:44:45.979338 (+ 3519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:45.980427 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:45.980432 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:45.980436 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:45.980590 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:45.981120 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:45.981125 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:45.981128 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:45.981453 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:44:45.981578 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 16:44:45.985374 (+ 3796us) server_negotiation.cc:299] Negotiation successful
0929 16:44:45.985641 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14425,"thread_start_us":14237,"threads_started":1}
I20260929 16:44:45.991189 14184 heartbeater.cc:347] Connected to a master server at 127.0.197.126:39101
I20260929 16:44:45.991703 14184 heartbeater.cc:464] Registering TS with master...
I20260929 16:44:45.992803 14184 heartbeater.cc:511] Master 127.0.197.126:39101 requested a full tablet report, sending...
I20260929 16:44:45.995383 13709 ts_manager.cc:194] Registered new tserver with Master: ccb1dba7c1d344698180c2109a98b49c (127.0.197.67:34281)
I20260929 16:44:45.996165 789 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:44:45.996486 13709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.67@KRBTEST.COM'} at 127.0.197.67:52641
I20260929 16:44:46.043493 14190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:46.004499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43160 (local address 127.0.197.126:39101)
0929 16:44:46.010016 (+ 5517us) server_negotiation.cc:206] Beginning negotiation
0929 16:44:46.010024 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:44:46.017105 (+ 7081us) server_negotiation.cc:407] Connection header received
0929 16:44:46.017446 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:44:46.017454 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:44:46.017565 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:44:46.017649 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:44:46.020470 (+ 2821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:46.021603 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:46.023823 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:46.024451 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:46.024644 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:46.030276 (+ 5632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:44:46.030285 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:44:46.030291 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:44:46.030346 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:44:46.034958 (+ 4612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:46.036746 (+ 1788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:46.036754 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:46.036758 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:46.036967 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:46.037539 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:46.037545 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:46.037548 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:46.038001 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:44:46.041698 (+ 3697us) server_negotiation.cc:1035] Waiting for connection context
0929 16:44:46.042720 (+ 1022us) server_negotiation.cc:299] Negotiation successful
0929 16:44:46.043107 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5246}
I20260929 16:44:46.053611 13709 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43160:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:44:46.066694 13709 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:44:46.138029 14205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:46.102529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.67:34281 (local address 127.0.0.1:47196)
0929 16:44:46.120880 (+ 18351us) negotiation.cc:107] Waiting for socket to connect
0929 16:44:46.120929 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 16:44:46.121150 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:44:46.125389 (+ 4239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:44:46.125400 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:44:46.125446 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:44:46.126046 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:44:46.126065 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:44:46.129708 (+ 3643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:44:46.129714 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:44:46.132558 (+ 2844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:44:46.132576 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:44:46.136805 (+ 4229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:44:46.136814 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:44:46.137120 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:46.137191 (+ 71us) client_negotiation.cc:769] Sending connection context
0929 16:44:46.137346 (+ 155us) client_negotiation.cc:240] Negotiation successful
0929 16:44:46.137583 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18032,"thread_start_us":25569,"threads_started":1}
I20260929 16:44:46.139377 14201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:46.116589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:39989 (local address 127.0.0.1:39358)
0929 16:44:46.126716 (+ 10127us) negotiation.cc:107] Waiting for socket to connect
0929 16:44:46.126737 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:44:46.126853 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:44:46.130882 (+ 4029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:44:46.130887 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:44:46.130918 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:44:46.131374 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:44:46.131388 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:44:46.133745 (+ 2357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:44:46.133750 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:44:46.135899 (+ 2149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:44:46.135920 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:44:46.138312 (+ 2392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:44:46.138318 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:44:46.138546 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:46.138581 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 16:44:46.138680 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 16:44:46.138830 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9987,"mutex_wait_us":32,"thread_start_us":3800,"threads_started":1}
I20260929 16:44:46.139560 14202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:46.102892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47196 (local address 127.0.197.67:34281)
0929 16:44:46.122827 (+ 19935us) server_negotiation.cc:206] Beginning negotiation
0929 16:44:46.122842 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:44:46.122912 (+ 70us) server_negotiation.cc:407] Connection header received
0929 16:44:46.124187 (+ 1275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:44:46.124202 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:44:46.124603 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:44:46.124917 (+ 314us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:44:46.127156 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:46.128791 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:46.132747 (+ 3956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:46.136213 (+ 3466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:46.136357 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:46.136616 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0929 16:44:46.138688 (+ 2072us) server_negotiation.cc:299] Negotiation successful
0929 16:44:46.138881 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19660,"thread_start_us":20910,"threads_started":1}
I20260929 16:44:46.141746 14206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:46.117221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39358 (local address 127.0.197.65:39989)
0929 16:44:46.130150 (+ 12929us) server_negotiation.cc:206] Beginning negotiation
0929 16:44:46.130160 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:44:46.130195 (+ 35us) server_negotiation.cc:407] Connection header received
0929 16:44:46.130304 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:44:46.130313 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:44:46.130612 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:44:46.130887 (+ 275us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:44:46.131586 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:46.133532 (+ 1946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:46.136156 (+ 2624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:46.137719 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:46.137873 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:46.138120 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0929 16:44:46.141146 (+ 3026us) server_negotiation.cc:299] Negotiation successful
0929 16:44:46.141374 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12708,"thread_start_us":13796,"threads_started":1}
I20260929 16:44:46.143808 14119 tablet_service.cc:1467] Processing CreateTablet for tablet da7b56d9f3f84d33bfa00bd14523a98f (DEFAULT_TABLE table=test-table [id=21f497e76871419ca5c89241c8bb9c51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:44:46.145581 14119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da7b56d9f3f84d33bfa00bd14523a98f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:44:46.146271 14203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:46.105412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.66:44971 (local address 127.0.0.1:60044)
0929 16:44:46.123521 (+ 18109us) negotiation.cc:107] Waiting for socket to connect
0929 16:44:46.123541 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:44:46.123709 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:44:46.129064 (+ 5355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:44:46.129071 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:44:46.129104 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:44:46.129475 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:44:46.129489 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:44:46.136857 (+ 7368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:44:46.136862 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:44:46.140829 (+ 3967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:44:46.140851 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:44:46.145175 (+ 4324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:44:46.145183 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:44:46.145427 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:46.145478 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 16:44:46.145599 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 16:44:46.145763 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17954,"mutex_wait_us":1324,"thread_start_us":17795,"threads_started":1}
I20260929 16:44:46.146512 13841 tablet_service.cc:1467] Processing CreateTablet for tablet da7b56d9f3f84d33bfa00bd14523a98f (DEFAULT_TABLE table=test-table [id=21f497e76871419ca5c89241c8bb9c51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:44:46.147380 14204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:46.105632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60044 (local address 127.0.197.66:44971)
0929 16:44:46.126953 (+ 21321us) server_negotiation.cc:206] Beginning negotiation
0929 16:44:46.126967 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:44:46.127039 (+ 72us) server_negotiation.cc:407] Connection header received
0929 16:44:46.127176 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:44:46.127189 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:44:46.127581 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:44:46.127947 (+ 366us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:44:46.135055 (+ 7108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:46.136685 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:46.143185 (+ 6500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:46.144673 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:46.144839 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:46.145047 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 16:44:46.146799 (+ 1752us) server_negotiation.cc:299] Negotiation successful
0929 16:44:46.147086 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21159,"thread_start_us":21057,"threads_started":1}
I20260929 16:44:46.148110 13841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da7b56d9f3f84d33bfa00bd14523a98f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:44:46.151226 13985 tablet_service.cc:1467] Processing CreateTablet for tablet da7b56d9f3f84d33bfa00bd14523a98f (DEFAULT_TABLE table=test-table [id=21f497e76871419ca5c89241c8bb9c51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:44:46.152531 13985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da7b56d9f3f84d33bfa00bd14523a98f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:44:46.180899 14207 tablet_bootstrap.cc:492] T da7b56d9f3f84d33bfa00bd14523a98f P ccb1dba7c1d344698180c2109a98b49c: Bootstrap starting.
I20260929 16:44:46.193078 14209 tablet_bootstrap.cc:492] T da7b56d9f3f84d33bfa00bd14523a98f P acac5948133b4015a4209e1058426c67: Bootstrap starting.
I20260929 16:44:46.193962 14208 tablet_bootstrap.cc:492] T da7b56d9f3f84d33bfa00bd14523a98f P 2e03bac8f6824f2ba8b3becabb46d105: Bootstrap starting.
I20260929 16:44:46.198180 14209 tablet_bootstrap.cc:654] T da7b56d9f3f84d33bfa00bd14523a98f P acac5948133b4015a4209e1058426c67: Neither blocks nor log segments found. Creating new log.
I20260929 16:44:46.202883 14207 tablet_bootstrap.cc:654] T da7b56d9f3f84d33bfa00bd14523a98f P ccb1dba7c1d344698180c2109a98b49c: Neither blocks nor log segments found. Creating new log.
I20260929 16:44:46.204468 14207 log.cc:824] T da7b56d9f3f84d33bfa00bd14523a98f P ccb1dba7c1d344698180c2109a98b49c: Log is configured to *not* fsync() on all Append() calls
I20260929 16:44:46.204711 14208 tablet_bootstrap.cc:654] T da7b56d9f3f84d33bfa00bd14523a98f P 2e03bac8f6824f2ba8b3becabb46d105: Neither blocks nor log segments found. Creating new log.
I20260929 16:44:46.206046 14208 log.cc:824] T da7b56d9f3f84d33bfa00bd14523a98f P 2e03bac8f6824f2ba8b3becabb46d105: Log is configured to *not* fsync() on all Append() calls
I20260929 16:44:46.200256 14050 heartbeater.cc:503] Master 127.0.197.126:39101 was elected leader, sending a full tablet report...
I20260929 16:44:46.209797 14209 log.cc:824] T da7b56d9f3f84d33bfa00bd14523a98f P acac5948133b4015a4209e1058426c67: Log is configured to *not* fsync() on all Append() calls
I20260929 16:44:46.224114 14209 tablet_bootstrap.cc:492] T da7b56d9f3f84d33bfa00bd14523a98f P acac5948133b4015a4209e1058426c67: No bootstrap required, opened a new log
I20260929 16:44:46.224644 14209 ts_tablet_manager.cc:1397] T da7b56d9f3f84d33bfa00bd14523a98f P acac5948133b4015a4209e1058426c67: Time spent bootstrapping tablet: real 0.032s user 0.009s sys 0.003s
I20260929 16:44:46.229352 14207 tablet_bootstrap.cc:492] T da7b56d9f3f84d33bfa00bd14523a98f P ccb1dba7c1d344698180c2109a98b49c: No bootstrap required, opened a new log
I20260929 16:44:46.230683 14208 tablet_bootstrap.cc:492] T da7b56d9f3f84d33bfa00bd14523a98f P 2e03bac8f6824f2ba8b3becabb46d105: No bootstrap required, opened a new log
I20260929 16:44:46.231163 14208 ts_tablet_manager.cc:1397] T da7b56d9f3f84d33bfa00bd14523a98f P 2e03bac8f6824f2ba8b3becabb46d105: Time spent bootstrapping tablet: real 0.037s user 0.012s sys 0.000s
I20260929 16:44:46.234229 14207 ts_tablet_manager.cc:1397] T da7b56d9f3f84d33bfa00bd14523a98f P ccb1dba7c1d344698180c2109a98b49c: Time spent bootstrapping tablet: real 0.054s user 0.010s sys 0.015s
I20260929 16:44:46.245896 14207 raft_consensus.cc:348] T da7b56d9f3f84d33bfa00bd14523a98f P ccb1dba7c1d344698180c2109a98b49c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb1dba7c1d344698180c2109a98b49c" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 34281 } } peers { permanent_uuid: "acac5948133b4015a4209e1058426c67" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44971 } } peers { permanent_uuid: "2e03bac8f6824f2ba8b3becabb46d105" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 39989 } }
I20260929 16:44:46.246626 14207 raft_consensus.cc:374] T da7b56d9f3f84d33bfa00bd14523a98f P ccb1dba7c1d344698180c2109a98b49c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:44:46.246783 14207 raft_consensus.cc:729] T da7b56d9f3f84d33bfa00bd14523a98f P ccb1dba7c1d344698180c2109a98b49c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccb1dba7c1d344698180c2109a98b49c, State: Initialized, Role: FOLLOWER
I20260929 16:44:46.247720 14207 consensus_queue.cc:260] T da7b56d9f3f84d33bfa00bd14523a98f P ccb1dba7c1d344698180c2109a98b49c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb1dba7c1d344698180c2109a98b49c" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 34281 } } peers { permanent_uuid: "acac5948133b4015a4209e1058426c67" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44971 } } peers { permanent_uuid: "2e03bac8f6824f2ba8b3becabb46d105" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 39989 } }
I20260929 16:44:46.253700 14209 raft_consensus.cc:348] T da7b56d9f3f84d33bfa00bd14523a98f P acac5948133b4015a4209e1058426c67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb1dba7c1d344698180c2109a98b49c" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 34281 } } peers { permanent_uuid: "acac5948133b4015a4209e1058426c67" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44971 } } peers { permanent_uuid: "2e03bac8f6824f2ba8b3becabb46d105" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 39989 } }
I20260929 16:44:46.254766 14208 raft_consensus.cc:348] T da7b56d9f3f84d33bfa00bd14523a98f P 2e03bac8f6824f2ba8b3becabb46d105 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb1dba7c1d344698180c2109a98b49c" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 34281 } } peers { permanent_uuid: "acac5948133b4015a4209e1058426c67" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44971 } } peers { permanent_uuid: "2e03bac8f6824f2ba8b3becabb46d105" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 39989 } }
I20260929 16:44:46.255255 14184 heartbeater.cc:503] Master 127.0.197.126:39101 was elected leader, sending a full tablet report...
I20260929 16:44:46.255347 14208 raft_consensus.cc:374] T da7b56d9f3f84d33bfa00bd14523a98f P 2e03bac8f6824f2ba8b3becabb46d105 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:44:46.255537 14208 raft_consensus.cc:729] T da7b56d9f3f84d33bfa00bd14523a98f P 2e03bac8f6824f2ba8b3becabb46d105 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e03bac8f6824f2ba8b3becabb46d105, State: Initialized, Role: FOLLOWER
I20260929 16:44:46.256413 14207 ts_tablet_manager.cc:1428] T da7b56d9f3f84d33bfa00bd14523a98f P ccb1dba7c1d344698180c2109a98b49c: Time spent starting tablet: real 0.022s user 0.014s sys 0.006s
I20260929 16:44:46.256419 14208 consensus_queue.cc:260] T da7b56d9f3f84d33bfa00bd14523a98f P 2e03bac8f6824f2ba8b3becabb46d105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb1dba7c1d344698180c2109a98b49c" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 34281 } } peers { permanent_uuid: "acac5948133b4015a4209e1058426c67" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44971 } } peers { permanent_uuid: "2e03bac8f6824f2ba8b3becabb46d105" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 39989 } }
I20260929 16:44:46.256875 14209 raft_consensus.cc:374] T da7b56d9f3f84d33bfa00bd14523a98f P acac5948133b4015a4209e1058426c67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:44:46.257058 14209 raft_consensus.cc:729] T da7b56d9f3f84d33bfa00bd14523a98f P acac5948133b4015a4209e1058426c67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acac5948133b4015a4209e1058426c67, State: Initialized, Role: FOLLOWER
I20260929 16:44:46.258239 14209 consensus_queue.cc:260] T da7b56d9f3f84d33bfa00bd14523a98f P acac5948133b4015a4209e1058426c67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb1dba7c1d344698180c2109a98b49c" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 34281 } } peers { permanent_uuid: "acac5948133b4015a4209e1058426c67" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44971 } } peers { permanent_uuid: "2e03bac8f6824f2ba8b3becabb46d105" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 39989 } }
I20260929 16:44:46.267295 14209 ts_tablet_manager.cc:1428] T da7b56d9f3f84d33bfa00bd14523a98f P acac5948133b4015a4209e1058426c67: Time spent starting tablet: real 0.042s user 0.015s sys 0.005s
I20260929 16:44:46.274271 13906 heartbeater.cc:503] Master 127.0.197.126:39101 was elected leader, sending a full tablet report...
I20260929 16:44:46.277009 14208 ts_tablet_manager.cc:1428] T da7b56d9f3f84d33bfa00bd14523a98f P 2e03bac8f6824f2ba8b3becabb46d105: Time spent starting tablet: real 0.046s user 0.012s sys 0.010s
I20260929 16:44:46.351097 14213 raft_consensus.cc:482] T da7b56d9f3f84d33bfa00bd14523a98f P ccb1dba7c1d344698180c2109a98b49c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:44:46.351420 14213 raft_consensus.cc:504] T da7b56d9f3f84d33bfa00bd14523a98f P ccb1dba7c1d344698180c2109a98b49c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb1dba7c1d344698180c2109a98b49c" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 34281 } } peers { permanent_uuid: "acac5948133b4015a4209e1058426c67" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44971 } } peers { permanent_uuid: "2e03bac8f6824f2ba8b3becabb46d105" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 39989 } }
I20260929 16:44:46.353240 14213 leader_election.cc:288] T da7b56d9f3f84d33bfa00bd14523a98f P ccb1dba7c1d344698180c2109a98b49c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers acac5948133b4015a4209e1058426c67 (127.0.197.66:44971), 2e03bac8f6824f2ba8b3becabb46d105 (127.0.197.65:39989)
I20260929 16:44:46.360881 14206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:46.353766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:38413 (local address 127.0.197.65:39989)
0929 16:44:46.354191 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0929 16:44:46.354197 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:44:46.354216 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:44:46.354299 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:44:46.354307 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:44:46.354382 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:44:46.354498 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:44:46.354945 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:46.356024 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:46.358681 (+ 2657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:46.359940 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:46.360134 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:46.360217 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 16:44:46.360482 (+ 265us) server_negotiation.cc:299] Negotiation successful
0929 16:44:46.360612 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283}
I20260929 16:44:46.360877 14189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:46.353559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:39989 (local address 127.0.197.67:38413)
0929 16:44:46.353768 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0929 16:44:46.353795 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 16:44:46.353959 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:44:46.354520 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:44:46.354525 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:44:46.354554 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:44:46.354766 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:44:46.354778 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:44:46.356273 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:44:46.356279 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:44:46.358486 (+ 2207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:44:46.358504 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:44:46.360141 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:44:46.360146 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:44:46.360316 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:46.360340 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 16:44:46.360450 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 16:44:46.360601 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 16:44:46.362397 14204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:46.353667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:37509 (local address 127.0.197.66:44971)
0929 16:44:46.354018 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0929 16:44:46.354025 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:44:46.356167 (+ 2142us) server_negotiation.cc:407] Connection header received
0929 16:44:46.356351 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:44:46.356355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:44:46.356418 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:44:46.356520 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:44:46.357097 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:46.357777 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:46.360107 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:46.361380 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:46.361555 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:46.361633 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 16:44:46.361993 (+ 360us) server_negotiation.cc:299] Negotiation successful
0929 16:44:46.362138 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178}
I20260929 16:44:46.362483 14223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:46.353967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.66:44971 (local address 127.0.197.67:37509)
0929 16:44:46.356036 (+ 2069us) negotiation.cc:107] Waiting for socket to connect
0929 16:44:46.356058 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:44:46.356191 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:44:46.356538 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:44:46.356543 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:44:46.356569 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:44:46.356940 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:44:46.356951 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:44:46.357978 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:44:46.357981 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:44:46.359877 (+ 1896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:44:46.359912 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:44:46.361630 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:44:46.361637 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:44:46.361821 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:46.361850 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 16:44:46.361964 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 16:44:46.362108 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1929,"mutex_wait_us":5,"thread_start_us":1742,"threads_started":1}
I20260929 16:44:46.362370 13861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da7b56d9f3f84d33bfa00bd14523a98f" candidate_uuid: "ccb1dba7c1d344698180c2109a98b49c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e03bac8f6824f2ba8b3becabb46d105" is_pre_election: true
I20260929 16:44:46.363046 13861 raft_consensus.cc:2454] T da7b56d9f3f84d33bfa00bd14523a98f P 2e03bac8f6824f2ba8b3becabb46d105 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccb1dba7c1d344698180c2109a98b49c in term 0.
I20260929 16:44:46.363890 14073 leader_election.cc:302] T da7b56d9f3f84d33bfa00bd14523a98f P ccb1dba7c1d344698180c2109a98b49c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e03bac8f6824f2ba8b3becabb46d105, ccb1dba7c1d344698180c2109a98b49c; no voters:
I20260929 16:44:46.363818 14005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da7b56d9f3f84d33bfa00bd14523a98f" candidate_uuid: "ccb1dba7c1d344698180c2109a98b49c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acac5948133b4015a4209e1058426c67" is_pre_election: true
I20260929 16:44:46.364398 14213 raft_consensus.cc:2781] T da7b56d9f3f84d33bfa00bd14523a98f P ccb1dba7c1d344698180c2109a98b49c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:44:46.364398 14005 raft_consensus.cc:2454] T da7b56d9f3f84d33bfa00bd14523a98f P acac5948133b4015a4209e1058426c67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccb1dba7c1d344698180c2109a98b49c in term 0.
I20260929 16:44:46.364558 14213 raft_consensus.cc:482] T da7b56d9f3f84d33bfa00bd14523a98f P ccb1dba7c1d344698180c2109a98b49c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:44:46.364657 14213 raft_consensus.cc:3037] T da7b56d9f3f84d33bfa00bd14523a98f P ccb1dba7c1d344698180c2109a98b49c [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:44:46.368003 14213 raft_consensus.cc:504] T da7b56d9f3f84d33bfa00bd14523a98f P ccb1dba7c1d344698180c2109a98b49c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb1dba7c1d344698180c2109a98b49c" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 34281 } } peers { permanent_uuid: "acac5948133b4015a4209e1058426c67" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44971 } } peers { permanent_uuid: "2e03bac8f6824f2ba8b3becabb46d105" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 39989 } }
I20260929 16:44:46.368870 14213 leader_election.cc:288] T da7b56d9f3f84d33bfa00bd14523a98f P ccb1dba7c1d344698180c2109a98b49c [CANDIDATE]: Term 1 election: Requested vote from peers acac5948133b4015a4209e1058426c67 (127.0.197.66:44971), 2e03bac8f6824f2ba8b3becabb46d105 (127.0.197.65:39989)
I20260929 16:44:46.369540 13861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da7b56d9f3f84d33bfa00bd14523a98f" candidate_uuid: "ccb1dba7c1d344698180c2109a98b49c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e03bac8f6824f2ba8b3becabb46d105"
I20260929 16:44:46.369544 14005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da7b56d9f3f84d33bfa00bd14523a98f" candidate_uuid: "ccb1dba7c1d344698180c2109a98b49c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acac5948133b4015a4209e1058426c67"
I20260929 16:44:46.369786 13861 raft_consensus.cc:3037] T da7b56d9f3f84d33bfa00bd14523a98f P 2e03bac8f6824f2ba8b3becabb46d105 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:44:46.369771 14005 raft_consensus.cc:3037] T da7b56d9f3f84d33bfa00bd14523a98f P acac5948133b4015a4209e1058426c67 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:44:46.372215 13861 raft_consensus.cc:2454] T da7b56d9f3f84d33bfa00bd14523a98f P 2e03bac8f6824f2ba8b3becabb46d105 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccb1dba7c1d344698180c2109a98b49c in term 1.
I20260929 16:44:46.372758 14073 leader_election.cc:302] T da7b56d9f3f84d33bfa00bd14523a98f P ccb1dba7c1d344698180c2109a98b49c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e03bac8f6824f2ba8b3becabb46d105, ccb1dba7c1d344698180c2109a98b49c; no voters:
I20260929 16:44:46.373099 14213 raft_consensus.cc:2781] T da7b56d9f3f84d33bfa00bd14523a98f P ccb1dba7c1d344698180c2109a98b49c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:44:46.373124 14005 raft_consensus.cc:2454] T da7b56d9f3f84d33bfa00bd14523a98f P acac5948133b4015a4209e1058426c67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccb1dba7c1d344698180c2109a98b49c in term 1.
I20260929 16:44:46.373690 14213 raft_consensus.cc:686] T da7b56d9f3f84d33bfa00bd14523a98f P ccb1dba7c1d344698180c2109a98b49c [term 1 LEADER]: Becoming Leader. State: Replica: ccb1dba7c1d344698180c2109a98b49c, State: Running, Role: LEADER
I20260929 16:44:46.374219 14213 consensus_queue.cc:237] T da7b56d9f3f84d33bfa00bd14523a98f P ccb1dba7c1d344698180c2109a98b49c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb1dba7c1d344698180c2109a98b49c" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 34281 } } peers { permanent_uuid: "acac5948133b4015a4209e1058426c67" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44971 } } peers { permanent_uuid: "2e03bac8f6824f2ba8b3becabb46d105" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 39989 } }
I20260929 16:44:46.379324 13709 catalog_manager.cc:5495] T da7b56d9f3f84d33bfa00bd14523a98f P ccb1dba7c1d344698180c2109a98b49c reported cstate change: term changed from 0 to 1, leader changed from <none> to ccb1dba7c1d344698180c2109a98b49c (127.0.197.67). New cstate: current_term: 1 leader_uuid: "ccb1dba7c1d344698180c2109a98b49c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb1dba7c1d344698180c2109a98b49c" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 34281 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "acac5948133b4015a4209e1058426c67" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e03bac8f6824f2ba8b3becabb46d105" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 39989 } health_report { overall_health: UNKNOWN } } }
W20260929 16:44:46.385251 13907 tablet.cc:2369] T da7b56d9f3f84d33bfa00bd14523a98f P 2e03bac8f6824f2ba8b3becabb46d105: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:44:46.411250 14051 tablet.cc:2369] T da7b56d9f3f84d33bfa00bd14523a98f P acac5948133b4015a4209e1058426c67: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:44:46.426055 14202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:46.420309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47204 (local address 127.0.197.67:34281)
0929 16:44:46.420513 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 16:44:46.420522 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:44:46.420607 (+ 85us) server_negotiation.cc:407] Connection header received
0929 16:44:46.420880 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:44:46.420884 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:44:46.420972 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:44:46.421137 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:44:46.421515 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:46.422262 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:46.423829 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:46.424238 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:46.424375 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:46.424870 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:44:46.425463 (+ 593us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:44:46.425635 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 16:44:46.425686 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 16:44:46.425776 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:44:46.435913 14005 raft_consensus.cc:1261] T da7b56d9f3f84d33bfa00bd14523a98f P acac5948133b4015a4209e1058426c67 [term 1 FOLLOWER]: Refusing update from remote peer ccb1dba7c1d344698180c2109a98b49c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:44:46.435913 13861 raft_consensus.cc:1261] T da7b56d9f3f84d33bfa00bd14523a98f P 2e03bac8f6824f2ba8b3becabb46d105 [term 1 FOLLOWER]: Refusing update from remote peer ccb1dba7c1d344698180c2109a98b49c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:44:46.437171 14224 consensus_queue.cc:1035] T da7b56d9f3f84d33bfa00bd14523a98f P ccb1dba7c1d344698180c2109a98b49c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e03bac8f6824f2ba8b3becabb46d105" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 39989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:44:46.437594 14213 consensus_queue.cc:1035] T da7b56d9f3f84d33bfa00bd14523a98f P ccb1dba7c1d344698180c2109a98b49c [LEADER]: Connected to new peer: Peer: permanent_uuid: "acac5948133b4015a4209e1058426c67" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:44:46.465775 13708 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:44:46.466183 13708 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:44:46.470814 13708 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9a77f6403a744551837636e865e63ada: Sending DeleteTablet for 3 replicas of tablet da7b56d9f3f84d33bfa00bd14523a98f
I20260929 16:44:46.472297 13985 tablet_service.cc:1514] Processing DeleteTablet for tablet da7b56d9f3f84d33bfa00bd14523a98f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:44:46 UTC) from {username='kudu', principal='kudu/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:60044
I20260929 16:44:46.472306 14119 tablet_service.cc:1514] Processing DeleteTablet for tablet da7b56d9f3f84d33bfa00bd14523a98f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:44:46 UTC) from {username='kudu', principal='kudu/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:47196
I20260929 16:44:46.472895 13841 tablet_service.cc:1514] Processing DeleteTablet for tablet da7b56d9f3f84d33bfa00bd14523a98f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:44:46 UTC) from {username='kudu', principal='kudu/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:39358
I20260929 16:44:46.473971 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 13746
I20260929 16:44:46.483098 14239 tablet_replica.cc:330] T da7b56d9f3f84d33bfa00bd14523a98f P 2e03bac8f6824f2ba8b3becabb46d105: stopping tablet replica
I20260929 16:44:46.483845 14239 raft_consensus.cc:2229] T da7b56d9f3f84d33bfa00bd14523a98f P 2e03bac8f6824f2ba8b3becabb46d105 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:44:46.484702 14239 raft_consensus.cc:2258] T da7b56d9f3f84d33bfa00bd14523a98f P 2e03bac8f6824f2ba8b3becabb46d105 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:44:46.487571 14239 ts_tablet_manager.cc:1905] T da7b56d9f3f84d33bfa00bd14523a98f P 2e03bac8f6824f2ba8b3becabb46d105: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:44:46.494763 14238 tablet_replica.cc:330] T da7b56d9f3f84d33bfa00bd14523a98f P acac5948133b4015a4209e1058426c67: stopping tablet replica
I20260929 16:44:46.495396 14238 raft_consensus.cc:2229] T da7b56d9f3f84d33bfa00bd14523a98f P acac5948133b4015a4209e1058426c67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:44:46.496022 14238 raft_consensus.cc:2258] T da7b56d9f3f84d33bfa00bd14523a98f P acac5948133b4015a4209e1058426c67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:44:46.496964 14237 tablet_replica.cc:330] T da7b56d9f3f84d33bfa00bd14523a98f P ccb1dba7c1d344698180c2109a98b49c: stopping tablet replica
I20260929 16:44:46.497426 14237 raft_consensus.cc:2229] T da7b56d9f3f84d33bfa00bd14523a98f P ccb1dba7c1d344698180c2109a98b49c [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:44:46.498232 14237 raft_consensus.cc:2258] T da7b56d9f3f84d33bfa00bd14523a98f P ccb1dba7c1d344698180c2109a98b49c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:44:46.498473 14239 ts_tablet_manager.cc:1918] T da7b56d9f3f84d33bfa00bd14523a98f P 2e03bac8f6824f2ba8b3becabb46d105: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:44:46.498716 14239 log.cc:1196] T da7b56d9f3f84d33bfa00bd14523a98f P 2e03bac8f6824f2ba8b3becabb46d105: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-0/wal/wals/da7b56d9f3f84d33bfa00bd14523a98f
I20260929 16:44:46.499518 14239 ts_tablet_manager.cc:1939] T da7b56d9f3f84d33bfa00bd14523a98f P 2e03bac8f6824f2ba8b3becabb46d105: Deleting consensus metadata
I20260929 16:44:46.500696 14238 ts_tablet_manager.cc:1905] T da7b56d9f3f84d33bfa00bd14523a98f P acac5948133b4015a4209e1058426c67: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:44:46.501792 13694 catalog_manager.cc:4841] TS 2e03bac8f6824f2ba8b3becabb46d105 (127.0.197.65:39989): tablet da7b56d9f3f84d33bfa00bd14523a98f (table test-table [id=21f497e76871419ca5c89241c8bb9c51]) successfully deleted
I20260929 16:44:46.502080 14206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:46.475701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39360 (local address 127.0.197.65:39989)
0929 16:44:46.475888 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 16:44:46.475897 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:44:46.477295 (+ 1398us) server_negotiation.cc:407] Connection header received
0929 16:44:46.477398 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:44:46.477406 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:44:46.477531 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:44:46.477655 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:44:46.481215 (+ 3560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:46.482793 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:46.487202 (+ 4409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:46.487961 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:46.488094 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:46.491968 (+ 3874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:44:46.491990 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:44:46.492012 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:44:46.492072 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:44:46.496970 (+ 4898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:46.498093 (+ 1123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:46.498103 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:46.498108 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:46.498300 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:46.498989 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:46.498994 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:46.498997 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:46.499522 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:44:46.500064 (+ 542us) server_negotiation.cc:1035] Waiting for connection context
0929 16:44:46.501359 (+ 1295us) server_negotiation.cc:299] Negotiation successful
0929 16:44:46.501656 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 16:44:46.502830 13901 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:44:46.507345 14237 ts_tablet_manager.cc:1905] T da7b56d9f3f84d33bfa00bd14523a98f P ccb1dba7c1d344698180c2109a98b49c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:44:46.512225 14238 ts_tablet_manager.cc:1918] T da7b56d9f3f84d33bfa00bd14523a98f P acac5948133b4015a4209e1058426c67: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:44:46.512570 14238 log.cc:1196] T da7b56d9f3f84d33bfa00bd14523a98f P acac5948133b4015a4209e1058426c67: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-1/wal/wals/da7b56d9f3f84d33bfa00bd14523a98f
I20260929 16:44:46.513252 14238 ts_tablet_manager.cc:1939] T da7b56d9f3f84d33bfa00bd14523a98f P acac5948133b4015a4209e1058426c67: Deleting consensus metadata
I20260929 16:44:46.515972 13693 catalog_manager.cc:4841] TS acac5948133b4015a4209e1058426c67 (127.0.197.66:44971): tablet da7b56d9f3f84d33bfa00bd14523a98f (table test-table [id=21f497e76871419ca5c89241c8bb9c51]) successfully deleted
I20260929 16:44:46.520269 14237 ts_tablet_manager.cc:1918] T da7b56d9f3f84d33bfa00bd14523a98f P ccb1dba7c1d344698180c2109a98b49c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:44:46.520603 14237 log.cc:1196] T da7b56d9f3f84d33bfa00bd14523a98f P ccb1dba7c1d344698180c2109a98b49c: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790700068255611-789-0/minicluster-data/ts-2/wal/wals/da7b56d9f3f84d33bfa00bd14523a98f
I20260929 16:44:46.522082 14237 ts_tablet_manager.cc:1939] T da7b56d9f3f84d33bfa00bd14523a98f P ccb1dba7c1d344698180c2109a98b49c: Deleting consensus metadata
I20260929 16:44:46.525079 13696 catalog_manager.cc:4841] TS ccb1dba7c1d344698180c2109a98b49c (127.0.197.67:34281): tablet da7b56d9f3f84d33bfa00bd14523a98f (table test-table [id=21f497e76871419ca5c89241c8bb9c51]) successfully deleted
I20260929 16:44:46.632314 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 13746
I20260929 16:44:46.651607 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 13910
I20260929 16:44:46.668865 14204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:46.652580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60048 (local address 127.0.197.66:44971)
0929 16:44:46.652789 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 16:44:46.652795 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:44:46.652846 (+ 51us) server_negotiation.cc:407] Connection header received
0929 16:44:46.652915 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:44:46.652921 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:44:46.653022 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:44:46.653144 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:44:46.654659 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:46.656105 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:46.657488 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:46.658159 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:46.658306 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:46.661129 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:44:46.661152 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:44:46.661175 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:44:46.661228 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:44:46.665555 (+ 4327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:46.666403 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:46.666415 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:46.666419 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:46.666601 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:46.667068 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:46.667074 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:46.667078 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:46.667568 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:44:46.667895 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0929 16:44:46.668207 (+ 312us) server_negotiation.cc:299] Negotiation successful
0929 16:44:46.668448 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 16:44:46.669777 14045 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:44:46.764524 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 13910
I20260929 16:44:46.783846 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 14054
I20260929 16:44:46.803404 14202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:46.784660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47216 (local address 127.0.197.67:34281)
0929 16:44:46.784845 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 16:44:46.784853 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:44:46.784875 (+ 22us) server_negotiation.cc:407] Connection header received
0929 16:44:46.785117 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:44:46.785122 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:44:46.785191 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:44:46.785303 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:44:46.786654 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:46.787662 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:46.788987 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:46.789457 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:46.789613 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:46.796157 (+ 6544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:44:46.796171 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:44:46.796186 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:44:46.796239 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:44:46.800102 (+ 3863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:46.801041 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:46.801049 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:46.801052 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:46.801220 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:46.801643 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:46.801651 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:46.801659 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:46.802076 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:44:46.802423 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0929 16:44:46.802736 (+ 313us) server_negotiation.cc:299] Negotiation successful
0929 16:44:46.803042 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 16:44:46.804164 14179 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:44:46.926720 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 14054
I20260929 16:44:46.947284 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 13676
I20260929 16:44:46.948523 13739 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:44:47.068354 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 13676
I20260929 16:44:47.090853 789 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 16:44:47.427311 789 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 16:44:47.427515 789 mini_ranger.cc:66] Stopping Ranger...
I20260929 16:44:47.774264 789 mini_ranger.cc:68] Stopped Ranger
2026-09-29 16:44:47.774 UTC [12728] LOG: received smart shutdown request
2026-09-29 16:44:47.778 UTC [12728] LOG: background worker "logical replication launcher" (PID 12737) exited with exit code 1
2026-09-29 16:44:47.778 UTC [12732] LOG: shutting down
2026-09-29 16:44:47.966 UTC [12728] LOG: database system is shut down
2026-09-29T16:44:47Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (58886 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:44:48 dist-test-slave-l31f krb5kdc[14252](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:44:48 dist-test-slave-l31f krb5kdc[14252](info): set up 2 sockets
Sep 29 16:44:48 dist-test-slave-l31f krb5kdc[14252](info): commencing operation
krb5kdc: starting...
W20260929 16:44:49.235666 789 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/krb5kdc/test-user.keytab.
Sep 29 16:44:49 dist-test-slave-l31f krb5kdc[14252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:44:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:44:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.197.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
I20260929 16:44:49.487454 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:33035
--webserver_interface=127.0.197.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:35745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.126:33035,127.0.197.125:36513,127.0.197.124:43947
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:44:49.837327 14265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:44:49.837744 14265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:44:49.837909 14265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:44:49.845841 14265 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:44:49.846036 14265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:44:49.846169 14265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:44:49.846275 14265 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:44:49.846359 14265 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:44:49.857069 14265 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:35745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.197.126:33035,127.0.197.125:36513,127.0.197.124:43947
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:33035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.197.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:44:49.858623 14265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:44:49.860183 14265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:44:49.874251 14271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:44:49.874274 14270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:44:49.874315 14273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:44:49.875254 14265 server_base.cc:1034] running on GCE node
I20260929 16:44:49.876259 14265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:44:49.878029 14265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:44:49.879311 14265 hybrid_clock.cc:648] HybridClock initialized: now 1790700289879262 us; error 49 us; skew 500 ppm
I20260929 16:44:49.879767 14265 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:44:49 dist-test-slave-l31f krb5kdc[14252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:44:49.883284 14265 init.cc:377] Logged in from keytab as kudu/127.0.197.126@KRBTEST.COM (short username kudu)
I20260929 16:44:49.885006 14265 webserver.cc:460] Webserver started at http://127.0.197.126:46255/ using document root <none> and password file <none>
I20260929 16:44:49.885735 14265 fs_manager.cc:359] Metadata directory not provided
I20260929 16:44:49.885843 14265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:44:49.886116 14265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:44:49.889343 14265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/minicluster-data/master-0/data/instance:
uuid: "bc33c23210eb482e98b568e5cc8ccdfe"
format_stamp: "Formatted at 2026-09-29 16:44:49 on dist-test-slave-l31f"
server_key: "215522ab56af8fee4a000cea133c7578"
server_key_iv: "8c683555f5a293d7581ddee6e77561f3"
server_key_version: "encryptionkey@0"
I20260929 16:44:49.890198 14265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/minicluster-data/master-0/wal/instance:
uuid: "bc33c23210eb482e98b568e5cc8ccdfe"
format_stamp: "Formatted at 2026-09-29 16:44:49 on dist-test-slave-l31f"
server_key: "215522ab56af8fee4a000cea133c7578"
server_key_iv: "8c683555f5a293d7581ddee6e77561f3"
server_key_version: "encryptionkey@0"
I20260929 16:44:49.896025 14265 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 16:44:49.899883 14280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:44:49.901556 14265 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:44:49.901757 14265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "bc33c23210eb482e98b568e5cc8ccdfe"
format_stamp: "Formatted at 2026-09-29 16:44:49 on dist-test-slave-l31f"
server_key: "215522ab56af8fee4a000cea133c7578"
server_key_iv: "8c683555f5a293d7581ddee6e77561f3"
server_key_version: "encryptionkey@0"
I20260929 16:44:49.901976 14265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:44:49.924808 14265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:44:49.929677 14265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:44:49.930017 14265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:44:49.950466 14265 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.126:33035
I20260929 16:44:49.950505 14331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.126:33035 every 8 connection(s)
I20260929 16:44:49.952103 14265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:44:49.956419 14332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:44:49.962541 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 14265
I20260929 16:44:49.962795 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/minicluster-data/master-0/wal/instance
I20260929 16:44:49.963541 789 external_mini_cluster.cc:1428] Setting key 0b7f08817c85a5c4602a26c039165f52
I20260929 16:44:49.964048 14332 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33035 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:44:49 dist-test-slave-l31f krb5kdc[14252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.197.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.125@KRBTEST.COM" created.
I20260929 16:44:49.990264 14334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:49.967750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:33035 (local address 127.0.0.1:38194)
0929 16:44:49.968910 (+ 1160us) negotiation.cc:107] Waiting for socket to connect
0929 16:44:49.968981 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0929 16:44:49.970996 (+ 2015us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:44:49.971972 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:44:49.971978 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:44:49.972752 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:44:49.973094 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:44:49.973114 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:44:49.974936 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:44:49.974941 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:44:49.976152 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:44:49.976171 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:44:49.977180 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:44:49.977187 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:44:49.977356 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:49.978172 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:44:49.978202 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:44:49.981335 (+ 3133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:44:49.985183 (+ 3848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:44:49.985194 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:44:49.985205 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:44:49.985740 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:44:49.986207 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:44:49.986211 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:44:49.986213 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:44:49.986369 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:44:49.987442 (+ 1073us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:44:49.987453 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:44:49.987776 (+ 323us) client_negotiation.cc:769] Sending connection context
0929 16:44:49.988113 (+ 337us) client_negotiation.cc:240] Negotiation successful
0929 16:44:49.988782 (+ 669us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":790,"mutex_wait_us":201,"thread_start_us":489,"threads_started":1}
I20260929 16:44:49.990280 14335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:49.967750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38194 (local address 127.0.197.126:33035)
0929 16:44:49.969671 (+ 1921us) server_negotiation.cc:206] Beginning negotiation
0929 16:44:49.969684 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:44:49.969745 (+ 61us) server_negotiation.cc:407] Connection header received
0929 16:44:49.971492 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:44:49.971498 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:44:49.971746 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:44:49.971971 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:44:49.973307 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:49.974740 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:49.976319 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:49.976932 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:49.977151 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:49.981649 (+ 4498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:44:49.981658 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:44:49.981673 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:44:49.981713 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:44:49.985016 (+ 3303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:49.985907 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:49.985914 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:49.985916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:49.986092 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:49.986469 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:49.986472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:49.986474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:49.987261 (+ 787us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:44:49.987454 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0929 16:44:49.988269 (+ 815us) server_negotiation.cc:299] Negotiation successful
0929 16:44:49.988655 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":1671,"spinlock_wait_cycles":38272,"thread_start_us":1449,"threads_started":1}
I20260929 16:44:49.993420 14332 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.197.125" port: 36513 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:44:49.994765 14334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:49.994191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.125:36513 (local address 127.0.0.1:35328)
0929 16:44:49.994411 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 16:44:49.994582 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.125:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260929 16:44:49.995129 14281 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.197.125:36513: connect: Connection refused (error 111)
I20260929 16:44:49.996836 14334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:49.996452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.125:36513 (local address 127.0.0.1:35342)
0929 16:44:49.996611 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 16:44:49.996691 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.125:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 16:44:49.997089 14332 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.197.125:36513: Network error: Client connection negotiation failed: client connection to 127.0.197.125:36513: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/krb5kdc/kudu_127.0.197.125.keytab.
Entry for principal kudu/127.0.197.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/krb5kdc/kudu_127.0.197.125.keytab.
I20260929 16:44:50.020706 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.125:36513
--webserver_interface=127.0.197.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/krb5kdc/kudu_127.0.197.125.keytab
--principal=kudu/127.0.197.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:35745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.126:33035,127.0.197.125:36513,127.0.197.124:43947
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:44:50.046566 14332 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.197.125" port: 36513 } attempt: 1
I20260929 16:44:50.047925 14334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:50.047442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.125:36513 (local address 127.0.0.1:35350)
0929 16:44:50.047655 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0929 16:44:50.047781 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.125:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260929 16:44:50.049086 14334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:50.048676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.125:36513 (local address 127.0.0.1:35356)
0929 16:44:50.048843 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 16:44:50.048916 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.125:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 16:44:50.049336 14332 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.197.125:36513: Network error: Client connection negotiation failed: client connection to 127.0.197.125:36513: connect: Connection refused (error 111)
I20260929 16:44:50.117628 14332 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.197.125" port: 36513 } attempt: 2
I20260929 16:44:50.119165 14334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:50.118509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.125:36513 (local address 127.0.0.1:35372)
0929 16:44:50.118774 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0929 16:44:50.118918 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.125:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260929 16:44:50.120374 14334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:50.119950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.125:36513 (local address 127.0.0.1:35384)
0929 16:44:50.120109 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 16:44:50.120221 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.125:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 16:44:50.120558 14332 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.197.125:36513: Network error: Client connection negotiation failed: client connection to 127.0.197.125:36513: connect: Connection refused (error 111)
I20260929 16:44:50.211822 14332 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.197.125" port: 36513 } attempt: 3
I20260929 16:44:50.213129 14334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:50.212636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.125:36513 (local address 127.0.0.1:35398)
0929 16:44:50.212873 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0929 16:44:50.212989 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.125:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260929 16:44:50.214308 14334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:50.213893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.125:36513 (local address 127.0.0.1:35410)
0929 16:44:50.214037 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 16:44:50.214138 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.125:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260929 16:44:50.214578 14332 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.197.125:36513: Network error: Client connection negotiation failed: client connection to 127.0.197.125:36513: connect: Connection refused (error 111)
I20260929 16:44:50.357924 14332 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.197.125" port: 36513 } attempt: 4
I20260929 16:44:50.359332 14334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:50.358686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.125:36513 (local address 127.0.0.1:35420)
0929 16:44:50.359090 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0929 16:44:50.359182 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.125:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170}
I20260929 16:44:50.360378 14334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:50.360000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.125:36513 (local address 127.0.0.1:35426)
0929 16:44:50.360152 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 16:44:50.360248 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.125:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 16:44:50.360598 14332 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.197.125:36513: Network error: Client connection negotiation failed: client connection to 127.0.197.125:36513: connect: Connection refused (error 111)
W20260929 16:44:50.377303 14339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:44:50.377640 14339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:44:50.377816 14339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:44:50.386225 14339 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:44:50.386371 14339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:44:50.386456 14339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:44:50.386565 14339 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:44:50.386629 14339 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:44:50.397194 14339 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:35745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.197.126:33035,127.0.197.125:36513,127.0.197.124:43947
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/krb5kdc/kudu_127.0.197.125.keytab
--principal=kudu/127.0.197.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.125:36513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.197.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:44:50.398707 14339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:44:50.400215 14339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:44:50.413043 14344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:44:50.413058 14345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:44:50.412951 14347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:44:50.414312 14339 server_base.cc:1034] running on GCE node
I20260929 16:44:50.415393 14339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:44:50.417120 14339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:44:50.418416 14339 hybrid_clock.cc:648] HybridClock initialized: now 1790700290418355 us; error 39 us; skew 500 ppm
I20260929 16:44:50.418833 14339 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:44:50 dist-test-slave-l31f krb5kdc[14252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700290, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:44:50.423214 14339 init.cc:377] Logged in from keytab as kudu/127.0.197.125@KRBTEST.COM (short username kudu)
I20260929 16:44:50.425024 14339 webserver.cc:460] Webserver started at http://127.0.197.125:41257/ using document root <none> and password file <none>
I20260929 16:44:50.425815 14339 fs_manager.cc:359] Metadata directory not provided
I20260929 16:44:50.425974 14339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:44:50.426303 14339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:44:50.429623 14339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/minicluster-data/master-1/data/instance:
uuid: "f524ae74c542424983640a068fd627cc"
format_stamp: "Formatted at 2026-09-29 16:44:50 on dist-test-slave-l31f"
server_key: "9e9a62d708bc08d4868242f9178bfe6c"
server_key_iv: "03a45f69b1a719c88898997cdea7d5c8"
server_key_version: "encryptionkey@0"
I20260929 16:44:50.430521 14339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/minicluster-data/master-1/wal/instance:
uuid: "f524ae74c542424983640a068fd627cc"
format_stamp: "Formatted at 2026-09-29 16:44:50 on dist-test-slave-l31f"
server_key: "9e9a62d708bc08d4868242f9178bfe6c"
server_key_iv: "03a45f69b1a719c88898997cdea7d5c8"
server_key_version: "encryptionkey@0"
I20260929 16:44:50.436352 14339 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 16:44:50.440234 14354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:44:50.441821 14339 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:44:50.442034 14339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/minicluster-data/master-1/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/minicluster-data/master-1/wal
uuid: "f524ae74c542424983640a068fd627cc"
format_stamp: "Formatted at 2026-09-29 16:44:50 on dist-test-slave-l31f"
server_key: "9e9a62d708bc08d4868242f9178bfe6c"
server_key_iv: "03a45f69b1a719c88898997cdea7d5c8"
server_key_version: "encryptionkey@0"
I20260929 16:44:50.442282 14339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:44:50.456111 14339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:44:50.461951 14339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:44:50.462327 14339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:44:50.483564 14339 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.125:36513
I20260929 16:44:50.483626 14405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.125:36513 every 8 connection(s)
I20260929 16:44:50.485148 14339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/minicluster-data/master-1/data/info.pb
I20260929 16:44:50.485774 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 14339
I20260929 16:44:50.486003 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/minicluster-data/master-1/wal/instance
I20260929 16:44:50.486691 789 external_mini_cluster.cc:1428] Setting key b4b048fd229622feaca868d33da1d446
I20260929 16:44:50.490891 14406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:44:50.500325 14406 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33035 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.197.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.124@KRBTEST.COM" created.
Sep 29 16:44:50 dist-test-slave-l31f krb5kdc[14252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700290, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.125@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:44:50.525585 14409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:50.503456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38196 (local address 127.0.197.126:33035)
0929 16:44:50.504020 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0929 16:44:50.504025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:44:50.505172 (+ 1147us) server_negotiation.cc:407] Connection header received
0929 16:44:50.507648 (+ 2476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:44:50.507654 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:44:50.507767 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:44:50.507905 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:44:50.509683 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:50.510560 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:50.512477 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:50.513185 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:50.513357 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:50.517666 (+ 4309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:44:50.517677 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:44:50.517685 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:44:50.517753 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:44:50.521321 (+ 3568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:50.522581 (+ 1260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:50.522597 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:50.522599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:50.522762 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:50.523488 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:50.523492 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:50.523495 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:50.523822 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:44:50.523956 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 16:44:50.525087 (+ 1131us) server_negotiation.cc:299] Negotiation successful
0929 16:44:50.525292 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":309,"threads_started":1}
I20260929 16:44:50.527247 14410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:50.504032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:33035 (local address 127.0.0.1:38196)
0929 16:44:50.504915 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0929 16:44:50.505038 (+ 123us) client_negotiation.cc:174] Beginning negotiation
0929 16:44:50.507312 (+ 2274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:44:50.508193 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:44:50.508203 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:44:50.508796 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:44:50.509473 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:44:50.509507 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:44:50.510756 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:44:50.510761 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:44:50.512295 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:44:50.512310 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:44:50.513486 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:44:50.513491 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:44:50.513716 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:50.514486 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:44:50.514512 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:44:50.517336 (+ 2824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:44:50.521563 (+ 4227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:44:50.521578 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:44:50.521603 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:44:50.522379 (+ 776us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:44:50.522969 (+ 590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:44:50.522978 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:44:50.522985 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:44:50.523321 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:44:50.524015 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:44:50.524027 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:44:50.524588 (+ 561us) client_negotiation.cc:769] Sending connection context
0929 16:44:50.524988 (+ 400us) client_negotiation.cc:240] Negotiation successful
0929 16:44:50.525623 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":399,"threads_started":1}
I20260929 16:44:50.529201 14406 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.197.125" port: 36513 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:44:50 dist-test-slave-l31f krb5kdc[14252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700290, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.125@KRBTEST.COM for kudu/127.0.197.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/krb5kdc/kudu_127.0.197.124.keytab.
Entry for principal kudu/127.0.197.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/krb5kdc/kudu_127.0.197.124.keytab.
I20260929 16:44:50.547353 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.124:43947
--webserver_interface=127.0.197.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/krb5kdc/kudu_127.0.197.124.keytab
--principal=kudu/127.0.197.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:35745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.126:33035,127.0.197.125:36513,127.0.197.124:43947
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:44:50.547828 14410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:50.529927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.125:36513 (local address 127.0.0.1:35432)
0929 16:44:50.530136 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0929 16:44:50.530155 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:44:50.530280 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:44:50.531464 (+ 1184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:44:50.531472 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:44:50.532066 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:44:50.532295 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:44:50.532311 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:44:50.533845 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:44:50.533854 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:44:50.535357 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:44:50.535375 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:44:50.536305 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:44:50.536314 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:44:50.536563 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:50.537239 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:44:50.537258 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:44:50.540681 (+ 3423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:44:50.544776 (+ 4095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:44:50.544781 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:44:50.544783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:44:50.545199 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:44:50.545684 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:44:50.545687 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:44:50.545690 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:44:50.545871 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:44:50.546699 (+ 828us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:44:50.546704 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:44:50.546859 (+ 155us) client_negotiation.cc:769] Sending connection context
0929 16:44:50.546997 (+ 138us) client_negotiation.cc:240] Negotiation successful
0929 16:44:50.547348 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 16:44:50.547887 14412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:50.530164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35432 (local address 127.0.197.125:36513)
0929 16:44:50.530965 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0929 16:44:50.530971 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:44:50.530996 (+ 25us) server_negotiation.cc:407] Connection header received
0929 16:44:50.531102 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:44:50.531108 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:44:50.531297 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:44:50.531446 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:44:50.532474 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:50.533685 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:50.535527 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:50.536133 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:50.536238 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:50.541011 (+ 4773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:44:50.541063 (+ 52us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:44:50.541080 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:44:50.541120 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:44:50.544566 (+ 3446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:50.545432 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:50.545440 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:50.545442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:50.545576 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:50.545991 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:50.545996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:50.545999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:50.546556 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:44:50.546796 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0929 16:44:50.547226 (+ 430us) server_negotiation.cc:299] Negotiation successful
0929 16:44:50.547445 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":488,"threads_started":1}
I20260929 16:44:50.550251 14406 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43947 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:44:50.551501 14410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:50.550995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43947 (local address 127.0.0.1:56950)
0929 16:44:50.551223 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0929 16:44:50.551340 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260929 16:44:50.551751 14355 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43947: connect: Connection refused (error 111)
I20260929 16:44:50.553479 14410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:50.553122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43947 (local address 127.0.0.1:56956)
0929 16:44:50.553280 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 16:44:50.553362 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 16:44:50.553747 14406 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.197.124:43947: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43947: connect: Connection refused (error 111)
I20260929 16:44:50.603185 14406 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43947 } attempt: 1
I20260929 16:44:50.604468 14410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:50.603979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43947 (local address 127.0.0.1:56960)
0929 16:44:50.604196 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0929 16:44:50.604307 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 16:44:50.605597 14410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:50.605208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43947 (local address 127.0.0.1:56972)
0929 16:44:50.605395 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 16:44:50.605470 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 16:44:50.605967 14406 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.197.124:43947: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43947: connect: Connection refused (error 111)
I20260929 16:44:50.659906 14332 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.197.125" port: 36513 } attempt: 5
Sep 29 16:44:50 dist-test-slave-l31f krb5kdc[14252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for kudu/127.0.197.125@KRBTEST.COM
I20260929 16:44:50.673794 14412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:50.661198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35446 (local address 127.0.197.125:36513)
0929 16:44:50.661427 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 16:44:50.661435 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:44:50.661455 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:44:50.661519 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:44:50.661523 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:44:50.661608 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:44:50.661720 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:44:50.662588 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:50.663378 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:50.664622 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:50.665146 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:50.665280 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:50.668472 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:44:50.668478 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:44:50.668481 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:44:50.668517 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:44:50.671151 (+ 2634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:50.671848 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:50.671853 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:50.671857 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:50.671986 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:50.672368 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:50.672373 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:50.672377 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:50.672675 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:44:50.672828 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 16:44:50.673224 (+ 396us) server_negotiation.cc:299] Negotiation successful
0929 16:44:50.673454 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 16:44:50.673794 14334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:50.660946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.125:36513 (local address 127.0.0.1:35446)
0929 16:44:50.661163 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0929 16:44:50.661188 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 16:44:50.661337 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:44:50.661765 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:44:50.661772 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:44:50.662216 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:44:50.662405 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:44:50.662416 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:44:50.663533 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:44:50.663539 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:44:50.664423 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:44:50.664435 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:44:50.665300 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:44:50.665306 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:44:50.665496 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:50.666153 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:44:50.666173 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:44:50.668251 (+ 2078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:44:50.671309 (+ 3058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:44:50.671314 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:44:50.671318 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:44:50.671723 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:44:50.672110 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:44:50.672114 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:44:50.672118 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:44:50.672260 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:44:50.672791 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:44:50.672796 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:44:50.672956 (+ 160us) client_negotiation.cc:769] Sending connection context
0929 16:44:50.673126 (+ 170us) client_negotiation.cc:240] Negotiation successful
0929 16:44:50.673428 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 16:44:50.674252 14406 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43947 } attempt: 2
I20260929 16:44:50.675179 14332 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43947 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:44:50.675242 14410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:50.674754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43947 (local address 127.0.0.1:56982)
0929 16:44:50.674942 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 16:44:50.675080 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 16:44:50.676189 14334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:50.675774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43947 (local address 127.0.0.1:56994)
0929 16:44:50.675910 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 16:44:50.676013 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260929 16:44:50.676656 14410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:50.676180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43947 (local address 127.0.0.1:57008)
0929 16:44:50.676370 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0929 16:44:50.676484 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260929 16:44:50.676965 14406 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.197.124:43947: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43947: connect: Connection refused (error 111)
I20260929 16:44:50.677117 14334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:50.676768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43947 (local address 127.0.0.1:57016)
0929 16:44:50.676924 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 16:44:50.676994 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 16:44:50.677282 14332 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.197.124:43947: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43947: connect: Connection refused (error 111)
I20260929 16:44:50.728552 14332 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43947 } attempt: 1
I20260929 16:44:50.729801 14334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:50.729301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43947 (local address 127.0.0.1:57024)
0929 16:44:50.729532 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 16:44:50.729630 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260929 16:44:50.730818 14334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:50.730468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43947 (local address 127.0.0.1:57038)
0929 16:44:50.730631 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 16:44:50.730704 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 16:44:50.731062 14332 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.197.124:43947: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43947: connect: Connection refused (error 111)
I20260929 16:44:50.768272 14406 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43947 } attempt: 3
I20260929 16:44:50.769861 14410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:50.769147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43947 (local address 127.0.0.1:57048)
0929 16:44:50.769462 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0929 16:44:50.769617 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260929 16:44:50.771481 14410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:50.770798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43947 (local address 127.0.0.1:57062)
0929 16:44:50.771148 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0929 16:44:50.771274 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
W20260929 16:44:50.771776 14406 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.197.124:43947: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43947: connect: Connection refused (error 111)
I20260929 16:44:50.799320 14332 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43947 } attempt: 2
I20260929 16:44:50.800560 14334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:50.800041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43947 (local address 127.0.0.1:57078)
0929 16:44:50.800281 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0929 16:44:50.800390 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 16:44:50.801579 14334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:50.801227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43947 (local address 127.0.0.1:57088)
0929 16:44:50.801374 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 16:44:50.801452 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 16:44:50.801832 14332 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.197.124:43947: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43947: connect: Connection refused (error 111)
W20260929 16:44:50.901140 14414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:44:50.901474 14414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:44:50.901646 14414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 16:44:50.908056 14332 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43947 } attempt: 3
I20260929 16:44:50.909168 14334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:50.908712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43947 (local address 127.0.0.1:57104)
0929 16:44:50.908909 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0929 16:44:50.909003 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260929 16:44:50.909327 14414 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:44:50.909471 14414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:44:50.909582 14414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:44:50.909688 14414 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:44:50.909758 14414 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:44:50.910252 14334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:50.909911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43947 (local address 127.0.0.1:57120)
0929 16:44:50.910056 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 16:44:50.910142 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 16:44:50.910492 14332 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.197.124:43947: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43947: connect: Connection refused (error 111)
I20260929 16:44:50.915073 14406 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43947 } attempt: 4
I20260929 16:44:50.916321 14410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:50.915733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43947 (local address 127.0.0.1:57128)
0929 16:44:50.915995 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 16:44:50.916118 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260929 16:44:50.917444 14410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:50.917053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43947 (local address 127.0.0.1:57134)
0929 16:44:50.917210 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 16:44:50.917302 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 16:44:50.917699 14406 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.197.124:43947: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43947: connect: Connection refused (error 111)
I20260929 16:44:50.922026 14414 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:35745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.197.126:33035,127.0.197.125:36513,127.0.197.124:43947
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/krb5kdc/kudu_127.0.197.124.keytab
--principal=kudu/127.0.197.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.124:43947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.197.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:44:50.923635 14414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:44:50.925043 14414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:44:50.938490 14422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:44:50.938571 14419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:44:50.938534 14420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:44:50.939627 14414 server_base.cc:1034] running on GCE node
I20260929 16:44:50.940527 14414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:44:50.942226 14414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:44:50.943491 14414 hybrid_clock.cc:648] HybridClock initialized: now 1790700290943436 us; error 51 us; skew 500 ppm
I20260929 16:44:50.943919 14414 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:44:50 dist-test-slave-l31f krb5kdc[14252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700290, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:44:50.947728 14414 init.cc:377] Logged in from keytab as kudu/127.0.197.124@KRBTEST.COM (short username kudu)
I20260929 16:44:50.949477 14414 webserver.cc:460] Webserver started at http://127.0.197.124:35515/ using document root <none> and password file <none>
I20260929 16:44:50.950449 14414 fs_manager.cc:359] Metadata directory not provided
I20260929 16:44:50.950595 14414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:44:50.950968 14414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:44:50.955487 14414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/minicluster-data/master-2/data/instance:
uuid: "dee2f370abc94d678c8d0307950b2e39"
format_stamp: "Formatted at 2026-09-29 16:44:50 on dist-test-slave-l31f"
server_key: "daf010a0f9f7d406f16aba4cb58a6ed5"
server_key_iv: "e941968b13940fa8a52759584c23cb1f"
server_key_version: "encryptionkey@0"
I20260929 16:44:50.956656 14414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/minicluster-data/master-2/wal/instance:
uuid: "dee2f370abc94d678c8d0307950b2e39"
format_stamp: "Formatted at 2026-09-29 16:44:50 on dist-test-slave-l31f"
server_key: "daf010a0f9f7d406f16aba4cb58a6ed5"
server_key_iv: "e941968b13940fa8a52759584c23cb1f"
server_key_version: "encryptionkey@0"
I20260929 16:44:50.963879 14414 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260929 16:44:50.967708 14429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:44:50.969404 14414 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:44:50.969586 14414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/minicluster-data/master-2/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/minicluster-data/master-2/wal
uuid: "dee2f370abc94d678c8d0307950b2e39"
format_stamp: "Formatted at 2026-09-29 16:44:50 on dist-test-slave-l31f"
server_key: "daf010a0f9f7d406f16aba4cb58a6ed5"
server_key_iv: "e941968b13940fa8a52759584c23cb1f"
server_key_version: "encryptionkey@0"
I20260929 16:44:50.969784 14414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:44:50.995299 14414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:44:51.001995 14414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:44:51.002369 14414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:44:51.023491 14414 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.124:43947
I20260929 16:44:51.023530 14480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.124:43947 every 8 connection(s)
I20260929 16:44:51.025106 14414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/minicluster-data/master-2/data/info.pb
I20260929 16:44:51.029280 14481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:44:51.032660 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 14414
I20260929 16:44:51.032902 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790700068255611-789-0/minicluster-data/master-2/wal/instance
I20260929 16:44:51.033421 789 external_mini_cluster.cc:1428] Setting key f0da3a8ad3ddfe2cdb4090669fa044ff
I20260929 16:44:51.038365 14481 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33035 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:44:51 dist-test-slave-l31f krb5kdc[14252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.124@KRBTEST.COM
Sep 29 16:44:51 dist-test-slave-l31f krb5kdc[14252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700290, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.124@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:44:51.058465 14485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:51.041149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38200 (local address 127.0.197.126:33035)
0929 16:44:51.041765 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0929 16:44:51.041771 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:44:51.042232 (+ 461us) server_negotiation.cc:407] Connection header received
0929 16:44:51.042463 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:44:51.042470 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:44:51.042583 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:44:51.042709 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:44:51.044003 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:51.044759 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:51.046443 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:51.047190 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:51.047345 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:51.051143 (+ 3798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:44:51.051151 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:44:51.051157 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:44:51.051201 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:44:51.054754 (+ 3553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:51.055899 (+ 1145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:51.055904 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:51.055906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:51.056131 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:51.056704 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:51.056711 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:51.056715 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:51.057070 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:44:51.057230 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 16:44:51.057779 (+ 549us) server_negotiation.cc:299] Negotiation successful
0929 16:44:51.058023 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":323,"threads_started":1}
I20260929 16:44:51.059542 14486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:51.041162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:33035 (local address 127.0.0.1:38200)
0929 16:44:51.042082 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0929 16:44:51.042122 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 16:44:51.042317 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:44:51.042783 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:44:51.042791 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:44:51.043552 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:44:51.043835 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:44:51.043844 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:44:51.044930 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:44:51.044940 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:44:51.046214 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:44:51.046233 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:44:51.047485 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:44:51.047491 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:44:51.047624 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:51.048306 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:44:51.048337 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:44:51.050873 (+ 2536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:44:51.054972 (+ 4099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:44:51.054989 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:44:51.055031 (+ 42us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:44:51.055726 (+ 695us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:44:51.056276 (+ 550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:44:51.056281 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:44:51.056303 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:44:51.056555 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:44:51.057200 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:44:51.057208 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:44:51.057425 (+ 217us) client_negotiation.cc:769] Sending connection context
0929 16:44:51.057699 (+ 274us) client_negotiation.cc:240] Negotiation successful
0929 16:44:51.058284 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":532,"threads_started":1}
I20260929 16:44:51.059559 14484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:51.035734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57136 (local address 127.0.197.124:43947)
0929 16:44:51.036628 (+ 894us) server_negotiation.cc:206] Beginning negotiation
0929 16:44:51.036641 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:44:51.036687 (+ 46us) server_negotiation.cc:407] Connection header received
0929 16:44:51.038433 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:44:51.038458 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:44:51.038938 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:44:51.039443 (+ 505us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:44:51.041062 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:51.043115 (+ 2053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:51.044602 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:51.045366 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:51.045598 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:51.050675 (+ 5077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:44:51.050684 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:44:51.050699 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:44:51.050737 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:44:51.054755 (+ 4018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:51.055885 (+ 1130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:51.055904 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:51.055910 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:51.056233 (+ 323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:51.056806 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:51.056812 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:51.056815 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:51.057383 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:44:51.057686 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0929 16:44:51.058055 (+ 369us) server_negotiation.cc:299] Negotiation successful
0929 16:44:51.058465 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":111,"server-negotiator.queue_time_us":659,"thread_start_us":397,"threads_started":1}
I20260929 16:44:51.061264 14481 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.197.125" port: 36513 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:44:51 dist-test-slave-l31f krb5kdc[14252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700290, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.124@KRBTEST.COM for kudu/127.0.197.125@KRBTEST.COM
Sep 29 16:44:51 dist-test-slave-l31f krb5kdc[14252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:44:51.074739 14486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:51.061985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.125:36513 (local address 127.0.0.1:35452)
0929 16:44:51.062234 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0929 16:44:51.062255 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:44:51.062360 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:44:51.062670 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:44:51.062674 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:44:51.063238 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:44:51.063425 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:44:51.063433 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:44:51.064395 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:44:51.064403 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:44:51.065277 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:44:51.065289 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:44:51.065977 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:44:51.065983 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:44:51.066198 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:51.066796 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:44:51.066815 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:44:51.069127 (+ 2312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:44:51.072220 (+ 3093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:44:51.072225 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:44:51.072229 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:44:51.072630 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:44:51.073015 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:44:51.073020 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:44:51.073032 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:44:51.073220 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:44:51.073723 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:44:51.073728 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:44:51.073909 (+ 181us) client_negotiation.cc:769] Sending connection context
0929 16:44:51.074057 (+ 148us) client_negotiation.cc:240] Negotiation successful
0929 16:44:51.074371 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138}
I20260929 16:44:51.075332 14412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:51.062164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35452 (local address 127.0.197.125:36513)
0929 16:44:51.062348 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 16:44:51.062354 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:44:51.062373 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:44:51.062468 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:44:51.062473 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:44:51.062547 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:44:51.062652 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:44:51.063566 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:51.064258 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:51.065417 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:51.065828 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:51.065955 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:51.069339 (+ 3384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:44:51.069344 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:44:51.069348 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:44:51.069383 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:44:51.072068 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:51.072755 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:51.072760 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:51.072764 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:51.072901 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:51.073320 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:51.073326 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:51.073330 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:51.073608 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:44:51.073772 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 16:44:51.074149 (+ 377us) server_negotiation.cc:299] Negotiation successful
0929 16:44:51.074532 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 16:44:51.076253 14481 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43947 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:44:51.077780 14485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:51.062890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38208 (local address 127.0.197.126:33035)
0929 16:44:51.063091 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 16:44:51.063095 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:44:51.063145 (+ 50us) server_negotiation.cc:407] Connection header received
0929 16:44:51.063346 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:44:51.063350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:44:51.063438 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:44:51.063548 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:44:51.064790 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:51.065520 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:51.066876 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:51.067360 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:51.067512 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:51.071832 (+ 4320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:44:51.071838 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:44:51.071841 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:44:51.071876 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:44:51.074702 (+ 2826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:51.075596 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:51.075602 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:51.075605 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:51.075768 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:51.076242 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:51.076247 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:51.076251 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:51.076604 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:44:51.076894 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0929 16:44:51.077179 (+ 285us) server_negotiation.cc:299] Negotiation successful
0929 16:44:51.077425 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Sep 29 16:44:51 dist-test-slave-l31f krb5kdc[14252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700290, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.124@KRBTEST.COM for kudu/127.0.197.124@KRBTEST.COM
I20260929 16:44:51.087697 14332 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43947 } attempt: 4
Sep 29 16:44:51 dist-test-slave-l31f krb5kdc[14252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.125@KRBTEST.COM
I20260929 16:44:51.091748 14486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:51.076972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43947 (local address 127.0.0.1:57152)
0929 16:44:51.077134 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 16:44:51.077155 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:44:51.077280 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:44:51.077791 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:44:51.077799 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:44:51.078206 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:44:51.078347 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:44:51.078356 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:44:51.079619 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:44:51.079624 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:44:51.080952 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:44:51.080968 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:44:51.081785 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:44:51.081793 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:44:51.082010 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:51.082694 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:44:51.082717 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:44:51.085173 (+ 2456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:44:51.088443 (+ 3270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:44:51.088447 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:44:51.088450 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:44:51.088963 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:44:51.089447 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:44:51.089463 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:44:51.089466 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:44:51.089758 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:44:51.090488 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:44:51.090493 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:44:51.090632 (+ 139us) client_negotiation.cc:769] Sending connection context
0929 16:44:51.090829 (+ 197us) client_negotiation.cc:240] Negotiation successful
0929 16:44:51.091265 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 16:44:51.092249 14484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:51.077351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57152 (local address 127.0.197.124:43947)
0929 16:44:51.077490 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 16:44:51.077494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:44:51.077506 (+ 12us) server_negotiation.cc:407] Connection header received
0929 16:44:51.077540 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:44:51.077543 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:44:51.077634 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:44:51.077740 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:44:51.078475 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:51.079485 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:51.081139 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:51.081617 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:51.081723 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:51.085375 (+ 3652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:44:51.085380 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:44:51.085384 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:44:51.085414 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:44:51.088290 (+ 2876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:51.089110 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:51.089117 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:51.089123 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:51.089306 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:51.089921 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:51.089927 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:51.089930 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:51.090362 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:44:51.090523 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 16:44:51.090955 (+ 432us) server_negotiation.cc:299] Negotiation successful
0929 16:44:51.091505 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":193,"server-negotiator.queue_time_us":58}
I20260929 16:44:51.097179 14412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:51.079869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35458 (local address 127.0.197.125:36513)
0929 16:44:51.080043 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 16:44:51.080048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:44:51.080062 (+ 14us) server_negotiation.cc:407] Connection header received
0929 16:44:51.080108 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:44:51.080112 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:44:51.080170 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:44:51.080249 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:44:51.081832 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:51.082938 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:51.084544 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:51.085035 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:51.085142 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:51.090437 (+ 5295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:44:51.090445 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:44:51.090449 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:44:51.090496 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:44:51.093911 (+ 3415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:51.094702 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:51.094710 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:51.094713 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:51.094869 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:51.095407 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:51.095414 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:51.095417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:51.095777 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:44:51.096081 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0929 16:44:51.096403 (+ 322us) server_negotiation.cc:299] Negotiation successful
0929 16:44:51.096648 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
Sep 29 16:44:51 dist-test-slave-l31f krb5kdc[14252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for kudu/127.0.197.124@KRBTEST.COM
I20260929 16:44:51.098901 14481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dee2f370abc94d678c8d0307950b2e39: Bootstrap starting.
I20260929 16:44:51.102741 789 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:44:51.103369 14481 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dee2f370abc94d678c8d0307950b2e39: Neither blocks nor log segments found. Creating new log.
I20260929 16:44:51.104727 14481 log.cc:824] T 00000000000000000000000000000000 P dee2f370abc94d678c8d0307950b2e39: Log is configured to *not* fsync() on all Append() calls
I20260929 16:44:51.108493 14481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dee2f370abc94d678c8d0307950b2e39: No bootstrap required, opened a new log
I20260929 16:44:51.109200 14334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:51.088428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43947 (local address 127.0.0.1:57168)
0929 16:44:51.088663 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0929 16:44:51.088684 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:44:51.088802 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:44:51.089861 (+ 1059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:44:51.089869 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:44:51.090363 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:44:51.090584 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:44:51.090599 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:44:51.092486 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:44:51.092492 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:44:51.093969 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:44:51.094000 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:44:51.095158 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:44:51.095164 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:44:51.095412 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:51.096014 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:44:51.096041 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:44:51.098627 (+ 2586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:44:51.106381 (+ 7754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:44:51.106387 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:44:51.106392 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:44:51.106989 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:44:51.107547 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:44:51.107554 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:44:51.107557 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:44:51.107740 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:44:51.108344 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:44:51.108349 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:44:51.108506 (+ 157us) client_negotiation.cc:769] Sending connection context
0929 16:44:51.108664 (+ 158us) client_negotiation.cc:240] Negotiation successful
0929 16:44:51.108921 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 16:44:51.110090 14487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:51.088738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57168 (local address 127.0.197.124:43947)
0929 16:44:51.089572 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0929 16:44:51.089576 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:44:51.089596 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:44:51.089634 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:44:51.089637 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:44:51.089689 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:44:51.089805 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:44:51.090781 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:51.092288 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:51.094203 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:51.094909 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:51.095135 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:51.098897 (+ 3762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:44:51.098904 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:44:51.098908 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:44:51.098948 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:44:51.102529 (+ 3581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:51.107175 (+ 4646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:51.107180 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:51.107182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:51.107331 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:51.107862 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:51.107868 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:51.107871 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:51.108199 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:44:51.108365 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 16:44:51.109604 (+ 1239us) server_negotiation.cc:299] Negotiation successful
0929 16:44:51.109787 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":413,"threads_started":1}
I20260929 16:44:51.116266 14332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc33c23210eb482e98b568e5cc8ccdfe: Bootstrap starting.
I20260929 16:44:51.119856 14481 raft_consensus.cc:348] T 00000000000000000000000000000000 P dee2f370abc94d678c8d0307950b2e39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc33c23210eb482e98b568e5cc8ccdfe" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33035 } } peers { permanent_uuid: "f524ae74c542424983640a068fd627cc" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 36513 } } peers { permanent_uuid: "dee2f370abc94d678c8d0307950b2e39" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43947 } }
I20260929 16:44:51.120443 14481 raft_consensus.cc:374] T 00000000000000000000000000000000 P dee2f370abc94d678c8d0307950b2e39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:44:51.120618 14481 raft_consensus.cc:729] T 00000000000000000000000000000000 P dee2f370abc94d678c8d0307950b2e39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dee2f370abc94d678c8d0307950b2e39, State: Initialized, Role: FOLLOWER
I20260929 16:44:51.121446 14481 consensus_queue.cc:260] T 00000000000000000000000000000000 P dee2f370abc94d678c8d0307950b2e39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc33c23210eb482e98b568e5cc8ccdfe" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33035 } } peers { permanent_uuid: "f524ae74c542424983640a068fd627cc" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 36513 } } peers { permanent_uuid: "dee2f370abc94d678c8d0307950b2e39" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43947 } }
I20260929 16:44:51.121840 14332 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc33c23210eb482e98b568e5cc8ccdfe: Neither blocks nor log segments found. Creating new log.
I20260929 16:44:51.123961 14332 log.cc:824] T 00000000000000000000000000000000 P bc33c23210eb482e98b568e5cc8ccdfe: Log is configured to *not* fsync() on all Append() calls
I20260929 16:44:51.128559 14495 sys_catalog.cc:455] T 00000000000000000000000000000000 P dee2f370abc94d678c8d0307950b2e39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc33c23210eb482e98b568e5cc8ccdfe" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33035 } } peers { permanent_uuid: "f524ae74c542424983640a068fd627cc" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 36513 } } peers { permanent_uuid: "dee2f370abc94d678c8d0307950b2e39" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43947 } } }
I20260929 16:44:51.129068 14495 sys_catalog.cc:458] T 00000000000000000000000000000000 P dee2f370abc94d678c8d0307950b2e39 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:44:51.131328 14481 sys_catalog.cc:564] T 00000000000000000000000000000000 P dee2f370abc94d678c8d0307950b2e39 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:44:51.137501 14485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:51.113056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38220 (local address 127.0.197.126:33035)
0929 16:44:51.113245 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 16:44:51.113251 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:44:51.115695 (+ 2444us) server_negotiation.cc:407] Connection header received
0929 16:44:51.116012 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:44:51.116018 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:44:51.116103 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:44:51.116177 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:44:51.118666 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:51.119923 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:51.122214 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:51.123327 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:51.123478 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:51.128918 (+ 5440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:44:51.128928 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:44:51.128933 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:44:51.128997 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:44:51.133103 (+ 4106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:51.134245 (+ 1142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:51.134252 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:51.134256 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:51.134434 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:51.134979 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:51.134985 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:51.134990 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:51.135440 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:44:51.135769 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0929 16:44:51.136603 (+ 834us) server_negotiation.cc:299] Negotiation successful
0929 16:44:51.136969 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 16:44:51.145707 14332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc33c23210eb482e98b568e5cc8ccdfe: No bootstrap required, opened a new log
I20260929 16:44:51.152055 14412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:51.116207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35466 (local address 127.0.197.125:36513)
0929 16:44:51.116396 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 16:44:51.116403 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:44:51.126399 (+ 9996us) server_negotiation.cc:407] Connection header received
0929 16:44:51.128382 (+ 1983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:44:51.128393 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:44:51.128514 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:44:51.128609 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:44:51.133942 (+ 5333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:51.135532 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:51.138677 (+ 3145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:51.139670 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:51.139809 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:51.144161 (+ 4352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:44:51.144176 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:44:51.144182 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:44:51.144250 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:44:51.148304 (+ 4054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:51.149328 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:51.149334 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:51.149339 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:51.149600 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:51.150070 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:51.150076 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:51.150079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:51.150506 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:44:51.150737 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0929 16:44:51.151405 (+ 668us) server_negotiation.cc:299] Negotiation successful
0929 16:44:51.151675 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 16:44:51.156137 14487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:51.127287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57178 (local address 127.0.197.124:43947)
0929 16:44:51.127465 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 16:44:51.127473 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:44:51.133332 (+ 5859us) server_negotiation.cc:407] Connection header received
0929 16:44:51.133684 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:44:51.133690 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:44:51.133794 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:44:51.133874 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:44:51.137428 (+ 3554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:51.138494 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:51.141056 (+ 2562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:51.141776 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:51.141892 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:51.146666 (+ 4774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:44:51.146679 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:44:51.146686 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:44:51.146749 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:44:51.151674 (+ 4925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:51.152887 (+ 1213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:51.152896 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:51.152902 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:51.153115 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:51.153679 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:51.153687 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:51.153693 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:51.154203 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:44:51.154507 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0929 16:44:51.155322 (+ 815us) server_negotiation.cc:299] Negotiation successful
0929 16:44:51.155623 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 16:44:51.158236 14332 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc33c23210eb482e98b568e5cc8ccdfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc33c23210eb482e98b568e5cc8ccdfe" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33035 } } peers { permanent_uuid: "f524ae74c542424983640a068fd627cc" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 36513 } } peers { permanent_uuid: "dee2f370abc94d678c8d0307950b2e39" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43947 } }
I20260929 16:44:51.158931 14332 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc33c23210eb482e98b568e5cc8ccdfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:44:51.159103 14332 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc33c23210eb482e98b568e5cc8ccdfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc33c23210eb482e98b568e5cc8ccdfe, State: Initialized, Role: FOLLOWER
I20260929 16:44:51.159755 14332 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc33c23210eb482e98b568e5cc8ccdfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc33c23210eb482e98b568e5cc8ccdfe" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33035 } } peers { permanent_uuid: "f524ae74c542424983640a068fd627cc" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 36513 } } peers { permanent_uuid: "dee2f370abc94d678c8d0307950b2e39" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43947 } }
I20260929 16:44:51.160794 14509 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc33c23210eb482e98b568e5cc8ccdfe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc33c23210eb482e98b568e5cc8ccdfe" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33035 } } peers { permanent_uuid: "f524ae74c542424983640a068fd627cc" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 36513 } } peers { permanent_uuid: "dee2f370abc94d678c8d0307950b2e39" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43947 } } }
I20260929 16:44:51.163120 14509 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc33c23210eb482e98b568e5cc8ccdfe [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:44:51.162216 14332 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc33c23210eb482e98b568e5cc8ccdfe [sys.catalog]: configured and running, proceeding with master startup.
W20260929 16:44:51.172227 14508 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dee2f370abc94d678c8d0307950b2e39: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:44:51.172389 14508 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:44:51.174096 14509 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc33c23210eb482e98b568e5cc8ccdfe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:44:51.174431 14509 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc33c23210eb482e98b568e5cc8ccdfe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc33c23210eb482e98b568e5cc8ccdfe" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33035 } } peers { permanent_uuid: "f524ae74c542424983640a068fd627cc" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 36513 } } peers { permanent_uuid: "dee2f370abc94d678c8d0307950b2e39" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43947 } }
I20260929 16:44:51.161044 14414 master_runner.cc:186] Error getting master registration for 127.0.197.126:33035: OK, instance_id { permanent_uuid: "bc33c23210eb482e98b568e5cc8ccdfe" instance_seqno: 1790700289944615 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 16:44:51.177044 14414 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 16:44:51.178646 14509 leader_election.cc:288] T 00000000000000000000000000000000 P bc33c23210eb482e98b568e5cc8ccdfe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f524ae74c542424983640a068fd627cc (127.0.197.125:36513), dee2f370abc94d678c8d0307950b2e39 (127.0.197.124:43947)
I20260929 16:44:51.180555 14456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc33c23210eb482e98b568e5cc8ccdfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dee2f370abc94d678c8d0307950b2e39" is_pre_election: true
I20260929 16:44:51.181212 14456 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dee2f370abc94d678c8d0307950b2e39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc33c23210eb482e98b568e5cc8ccdfe in term 0.
I20260929 16:44:51.181942 14281 leader_election.cc:302] T 00000000000000000000000000000000 P bc33c23210eb482e98b568e5cc8ccdfe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc33c23210eb482e98b568e5cc8ccdfe, dee2f370abc94d678c8d0307950b2e39; no voters:
I20260929 16:44:51.182456 14509 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc33c23210eb482e98b568e5cc8ccdfe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:44:51.182677 14509 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc33c23210eb482e98b568e5cc8ccdfe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:44:51.182809 14509 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc33c23210eb482e98b568e5cc8ccdfe [term 0 FOLLOWER]: Advancing to term 1
W20260929 16:44:51.183442 14286 tablet.cc:2369] T 00000000000000000000000000000000 P bc33c23210eb482e98b568e5cc8ccdfe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:44:51.188300 14521 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 16:44:51.189193 14509 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc33c23210eb482e98b568e5cc8ccdfe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc33c23210eb482e98b568e5cc8ccdfe" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33035 } } peers { permanent_uuid: "f524ae74c542424983640a068fd627cc" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 36513 } } peers { permanent_uuid: "dee2f370abc94d678c8d0307950b2e39" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43947 } }
I20260929 16:44:51.189734 14509 leader_election.cc:288] T 00000000000000000000000000000000 P bc33c23210eb482e98b568e5cc8ccdfe [CANDIDATE]: Term 1 election: Requested vote from peers f524ae74c542424983640a068fd627cc (127.0.197.125:36513), dee2f370abc94d678c8d0307950b2e39 (127.0.197.124:43947)
I20260929 16:44:51.190372 14456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc33c23210eb482e98b568e5cc8ccdfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dee2f370abc94d678c8d0307950b2e39"
I20260929 16:44:51.190619 14456 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dee2f370abc94d678c8d0307950b2e39 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:44:51.194262 14456 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dee2f370abc94d678c8d0307950b2e39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc33c23210eb482e98b568e5cc8ccdfe in term 1.
I20260929 16:44:51.194733 14281 leader_election.cc:302] T 00000000000000000000000000000000 P bc33c23210eb482e98b568e5cc8ccdfe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc33c23210eb482e98b568e5cc8ccdfe, dee2f370abc94d678c8d0307950b2e39; no voters:
I20260929 16:44:51.195145 14509 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc33c23210eb482e98b568e5cc8ccdfe [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:44:51.195864 14509 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc33c23210eb482e98b568e5cc8ccdfe [term 1 LEADER]: Becoming Leader. State: Replica: bc33c23210eb482e98b568e5cc8ccdfe, State: Running, Role: LEADER
I20260929 16:44:51.196555 14509 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc33c23210eb482e98b568e5cc8ccdfe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc33c23210eb482e98b568e5cc8ccdfe" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33035 } } peers { permanent_uuid: "f524ae74c542424983640a068fd627cc" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 36513 } } peers { permanent_uuid: "dee2f370abc94d678c8d0307950b2e39" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43947 } }
I20260929 16:44:51.216914 14406 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43947 } attempt: 5
I20260929 16:44:51.193311 14265 master_runner.cc:186] Error getting master registration for 127.0.197.125:36513: OK, instance_id { permanent_uuid: "f524ae74c542424983640a068fd627cc" instance_seqno: 1790700290477539 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 16:44:51.222561 14265 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 16:44:51.224123 14522 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc33c23210eb482e98b568e5cc8ccdfe [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc33c23210eb482e98b568e5cc8ccdfe. Latest consensus state: current_term: 1 leader_uuid: "bc33c23210eb482e98b568e5cc8ccdfe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc33c23210eb482e98b568e5cc8ccdfe" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33035 } } peers { permanent_uuid: "f524ae74c542424983640a068fd627cc" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 36513 } } peers { permanent_uuid: "dee2f370abc94d678c8d0307950b2e39" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43947 } } }
I20260929 16:44:51.224557 14522 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc33c23210eb482e98b568e5cc8ccdfe [sys.catalog]: This master's current role is: LEADER
I20260929 16:44:51.225783 14524 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 29 16:44:51 dist-test-slave-l31f krb5kdc[14252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700290, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.125@KRBTEST.COM for kudu/127.0.197.124@KRBTEST.COM
I20260929 16:44:51.232758 14524 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:44:51.240787 14410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:51.218544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43947 (local address 127.0.0.1:57180)
0929 16:44:51.218872 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0929 16:44:51.218893 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:44:51.219061 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:44:51.221542 (+ 2481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:44:51.221548 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:44:51.222127 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:44:51.222380 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:44:51.222396 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:44:51.224316 (+ 1920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:44:51.224323 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:44:51.226097 (+ 1774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:44:51.226120 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:44:51.227565 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:44:51.227573 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:44:51.227827 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:51.228720 (+ 893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:44:51.228754 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:44:51.232529 (+ 3775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:44:51.237405 (+ 4876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:44:51.237412 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:44:51.237416 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:44:51.238062 (+ 646us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:44:51.238654 (+ 592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:44:51.238660 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:44:51.238664 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:44:51.238871 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:44:51.239637 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:44:51.239642 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:44:51.239823 (+ 181us) client_negotiation.cc:769] Sending connection context
0929 16:44:51.240007 (+ 184us) client_negotiation.cc:240] Negotiation successful
0929 16:44:51.240350 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195}
I20260929 16:44:51.242712 14456 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dee2f370abc94d678c8d0307950b2e39 [term 1 FOLLOWER]: Refusing update from remote peer bc33c23210eb482e98b568e5cc8ccdfe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:44:51.243785 14522 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bc33c23210eb482e98b568e5cc8ccdfe [LEADER]: Connected to new peer: Peer: permanent_uuid: "dee2f370abc94d678c8d0307950b2e39" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:44:51.244119 14487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:51.220202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57180 (local address 127.0.197.124:43947)
0929 16:44:51.220405 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 16:44:51.220411 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:44:51.220433 (+ 22us) server_negotiation.cc:407] Connection header received
0929 16:44:51.220514 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:44:51.220521 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:44:51.220615 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:44:51.220737 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:44:51.222863 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:51.224107 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:51.226292 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:51.227105 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:51.227247 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:51.232960 (+ 5713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:44:51.232969 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:44:51.232975 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:44:51.233026 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:44:51.237204 (+ 4178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:51.238284 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:51.238289 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:51.238293 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:51.238485 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:51.239083 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:51.239089 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:51.239092 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:51.239482 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:44:51.241273 (+ 1791us) server_negotiation.cc:1035] Waiting for connection context
0929 16:44:51.242237 (+ 964us) server_negotiation.cc:299] Negotiation successful
0929 16:44:51.243690 (+ 1453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 16:44:51.250236 14406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f524ae74c542424983640a068fd627cc: Bootstrap starting.
W20260929 16:44:51.254774 14435 tablet.cc:2369] T 00000000000000000000000000000000 P dee2f370abc94d678c8d0307950b2e39: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:44:51.264246 14406 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f524ae74c542424983640a068fd627cc: Neither blocks nor log segments found. Creating new log.
I20260929 16:44:51.264618 14495 sys_catalog.cc:455] T 00000000000000000000000000000000 P dee2f370abc94d678c8d0307950b2e39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc33c23210eb482e98b568e5cc8ccdfe. Latest consensus state: current_term: 1 leader_uuid: "bc33c23210eb482e98b568e5cc8ccdfe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc33c23210eb482e98b568e5cc8ccdfe" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33035 } } peers { permanent_uuid: "f524ae74c542424983640a068fd627cc" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 36513 } } peers { permanent_uuid: "dee2f370abc94d678c8d0307950b2e39" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43947 } } }
I20260929 16:44:51.264950 14495 sys_catalog.cc:458] T 00000000000000000000000000000000 P dee2f370abc94d678c8d0307950b2e39 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:44:51.267899 14526 mvcc.cc:205] Tried to move back new op lower bound from 7334708392919015424 to 7334708392746328064. Current Snapshot: MvccSnapshot[applied={T|T < 7334708392919015424}]
I20260929 16:44:51.272732 14527 mvcc.cc:205] Tried to move back new op lower bound from 7334708392919015424 to 7334708392746328064. Current Snapshot: MvccSnapshot[applied={T|T < 7334708392919015424}]
I20260929 16:44:51.279633 14406 log.cc:824] T 00000000000000000000000000000000 P f524ae74c542424983640a068fd627cc: Log is configured to *not* fsync() on all Append() calls
I20260929 16:44:51.285804 14495 sys_catalog.cc:455] T 00000000000000000000000000000000 P dee2f370abc94d678c8d0307950b2e39 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bc33c23210eb482e98b568e5cc8ccdfe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc33c23210eb482e98b568e5cc8ccdfe" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33035 } } peers { permanent_uuid: "f524ae74c542424983640a068fd627cc" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 36513 } } peers { permanent_uuid: "dee2f370abc94d678c8d0307950b2e39" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43947 } } }
I20260929 16:44:51.286129 14495 sys_catalog.cc:458] T 00000000000000000000000000000000 P dee2f370abc94d678c8d0307950b2e39 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:44:51.289144 14509 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc33c23210eb482e98b568e5cc8ccdfe [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bc33c23210eb482e98b568e5cc8ccdfe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc33c23210eb482e98b568e5cc8ccdfe" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33035 } } peers { permanent_uuid: "f524ae74c542424983640a068fd627cc" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 36513 } } peers { permanent_uuid: "dee2f370abc94d678c8d0307950b2e39" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43947 } } }
I20260929 16:44:51.289516 14509 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc33c23210eb482e98b568e5cc8ccdfe [sys.catalog]: This master's current role is: LEADER
I20260929 16:44:51.293958 14524 catalog_manager.cc:1347] Generated new cluster ID: aba918d91bb84dd4b3e627ff70790a3a
I20260929 16:44:51.294096 14524 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:44:51.307123 14406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f524ae74c542424983640a068fd627cc: No bootstrap required, opened a new log
I20260929 16:44:51.333250 14406 raft_consensus.cc:348] T 00000000000000000000000000000000 P f524ae74c542424983640a068fd627cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc33c23210eb482e98b568e5cc8ccdfe" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33035 } } peers { permanent_uuid: "f524ae74c542424983640a068fd627cc" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 36513 } } peers { permanent_uuid: "dee2f370abc94d678c8d0307950b2e39" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43947 } }
I20260929 16:44:51.333968 14406 raft_consensus.cc:374] T 00000000000000000000000000000000 P f524ae74c542424983640a068fd627cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:44:51.334149 14406 raft_consensus.cc:729] T 00000000000000000000000000000000 P f524ae74c542424983640a068fd627cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f524ae74c542424983640a068fd627cc, State: Initialized, Role: FOLLOWER
I20260929 16:44:51.336859 14406 consensus_queue.cc:260] T 00000000000000000000000000000000 P f524ae74c542424983640a068fd627cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc33c23210eb482e98b568e5cc8ccdfe" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33035 } } peers { permanent_uuid: "f524ae74c542424983640a068fd627cc" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 36513 } } peers { permanent_uuid: "dee2f370abc94d678c8d0307950b2e39" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43947 } }
I20260929 16:44:51.345309 14524 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:44:51.347415 14524 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:44:51.357141 14406 sys_catalog.cc:564] T 00000000000000000000000000000000 P f524ae74c542424983640a068fd627cc [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:44:51.359194 14381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc33c23210eb482e98b568e5cc8ccdfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f524ae74c542424983640a068fd627cc" is_pre_election: true
I20260929 16:44:51.359602 14380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc33c23210eb482e98b568e5cc8ccdfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f524ae74c542424983640a068fd627cc"
I20260929 16:44:51.368678 14532 sys_catalog.cc:455] T 00000000000000000000000000000000 P f524ae74c542424983640a068fd627cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc33c23210eb482e98b568e5cc8ccdfe" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33035 } } peers { permanent_uuid: "f524ae74c542424983640a068fd627cc" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 36513 } } peers { permanent_uuid: "dee2f370abc94d678c8d0307950b2e39" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43947 } } }
I20260929 16:44:51.369171 14532 sys_catalog.cc:458] T 00000000000000000000000000000000 P f524ae74c542424983640a068fd627cc [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:44:51.369781 14379 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f524ae74c542424983640a068fd627cc [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:44:51.374275 14379 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f524ae74c542424983640a068fd627cc [term 1 FOLLOWER]: Refusing update from remote peer bc33c23210eb482e98b568e5cc8ccdfe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:44:51.376531 14522 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bc33c23210eb482e98b568e5cc8ccdfe [LEADER]: Connected to new peer: Peer: permanent_uuid: "f524ae74c542424983640a068fd627cc" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 36513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:44:51.394120 14524 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc33c23210eb482e98b568e5cc8ccdfe: Generated new TSK 0
I20260929 16:44:51.395515 14524 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:44:51.448233 14532 sys_catalog.cc:455] T 00000000000000000000000000000000 P f524ae74c542424983640a068fd627cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc33c23210eb482e98b568e5cc8ccdfe. Latest consensus state: current_term: 1 leader_uuid: "bc33c23210eb482e98b568e5cc8ccdfe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc33c23210eb482e98b568e5cc8ccdfe" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33035 } } peers { permanent_uuid: "f524ae74c542424983640a068fd627cc" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 36513 } } peers { permanent_uuid: "dee2f370abc94d678c8d0307950b2e39" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43947 } } }
I20260929 16:44:51.448222 14540 sys_catalog.cc:455] T 00000000000000000000000000000000 P f524ae74c542424983640a068fd627cc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bc33c23210eb482e98b568e5cc8ccdfe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc33c23210eb482e98b568e5cc8ccdfe" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33035 } } peers { permanent_uuid: "f524ae74c542424983640a068fd627cc" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 36513 } } peers { permanent_uuid: "dee2f370abc94d678c8d0307950b2e39" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43947 } } }
I20260929 16:44:51.448679 14532 sys_catalog.cc:458] T 00000000000000000000000000000000 P f524ae74c542424983640a068fd627cc [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:44:51.448680 14540 sys_catalog.cc:458] T 00000000000000000000000000000000 P f524ae74c542424983640a068fd627cc [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:44:51.448626 14509 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc33c23210eb482e98b568e5cc8ccdfe [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bc33c23210eb482e98b568e5cc8ccdfe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc33c23210eb482e98b568e5cc8ccdfe" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33035 } } peers { permanent_uuid: "f524ae74c542424983640a068fd627cc" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 36513 } } peers { permanent_uuid: "dee2f370abc94d678c8d0307950b2e39" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43947 } } }
I20260929 16:44:51.449023 14509 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc33c23210eb482e98b568e5cc8ccdfe [sys.catalog]: This master's current role is: LEADER
I20260929 16:44:51.490664 14551 catalog_manager.cc:1259] Loaded cluster ID: aba918d91bb84dd4b3e627ff70790a3a
I20260929 16:44:51.490872 14551 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f524ae74c542424983640a068fd627cc: loading cluster ID for follower catalog manager: success
I20260929 16:44:51.496065 14551 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f524ae74c542424983640a068fd627cc: acquiring CA information for follower catalog manager: success
I20260929 16:44:51.504793 14551 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f524ae74c542424983640a068fd627cc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 16:44:52.174707 14508 catalog_manager.cc:1259] Loaded cluster ID: aba918d91bb84dd4b3e627ff70790a3a
I20260929 16:44:52.174829 14508 catalog_manager.cc:1552] T 00000000000000000000000000000000 P dee2f370abc94d678c8d0307950b2e39: loading cluster ID for follower catalog manager: success
I20260929 16:44:52.177695 14508 catalog_manager.cc:1574] T 00000000000000000000000000000000 P dee2f370abc94d678c8d0307950b2e39: acquiring CA information for follower catalog manager: success
I20260929 16:44:52.179975 14508 catalog_manager.cc:1602] T 00000000000000000000000000000000 P dee2f370abc94d678c8d0307950b2e39: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 16:44:52.562289 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 14265
I20260929 16:44:52.564265 14327 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:44:52.678784 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 14265
I20260929 16:44:52.702045 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 14339
I20260929 16:44:52.703289 14401 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:44:52.812431 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 14339
I20260929 16:44:52.830760 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 14414
I20260929 16:44:52.832165 14476 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:44:52.944375 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 14414
2026-09-29T16:44:52Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4840 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790700068255611-789-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:44:53 dist-test-slave-l31f krb5kdc[14559](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:44:53 dist-test-slave-l31f krb5kdc[14559](info): set up 2 sockets
Sep 29 16:44:53 dist-test-slave-l31f krb5kdc[14559](info): commencing operation
krb5kdc: starting...
W20260929 16:44:54.079607 789 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790700068255611-789-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790700068255611-789-0/krb5kdc/test-user.keytab.
Sep 29 16:44:54 dist-test-slave-l31f krb5kdc[14559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:44:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:44:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.197.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
I20260929 16:44:54.339771 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:44385
--webserver_interface=127.0.197.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:35849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.126:44385
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:44:54.690057 14572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:44:54.690409 14572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:44:54.690559 14572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:44:54.698534 14572 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:44:54.698681 14572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:44:54.698771 14572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:44:54.698868 14572 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:44:54.698953 14572 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:44:54.709847 14572 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:35849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790700068255611-789-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.197.126:44385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:44385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.197.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:44:54.711447 14572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:44:54.712901 14572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:44:54.726347 14578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:44:54.726279 14580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:44:54.726321 14577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:44:54.727504 14572 server_base.cc:1034] running on GCE node
I20260929 16:44:54.728514 14572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:44:54.730137 14572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:44:54.731436 14572 hybrid_clock.cc:648] HybridClock initialized: now 1790700294731353 us; error 60 us; skew 500 ppm
I20260929 16:44:54.731868 14572 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:44:54 dist-test-slave-l31f krb5kdc[14559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700294, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:44:54.735658 14572 init.cc:377] Logged in from keytab as kudu/127.0.197.126@KRBTEST.COM (short username kudu)
I20260929 16:44:54.737455 14572 webserver.cc:460] Webserver started at http://127.0.197.126:44535/ using document root <none> and password file <none>
I20260929 16:44:54.738270 14572 fs_manager.cc:359] Metadata directory not provided
I20260929 16:44:54.738440 14572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:44:54.738804 14572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:44:54.742411 14572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790700068255611-789-0/minicluster-data/master-0/data/instance:
uuid: "ddf5cbc0146144f0b8e0519d3eab394a"
format_stamp: "Formatted at 2026-09-29 16:44:54 on dist-test-slave-l31f"
server_key: "fd524beb81fc214658f7d313773c83da"
server_key_iv: "372f716309d798e7ca6acd7d5872bd88"
server_key_version: "encryptionkey@0"
I20260929 16:44:54.743436 14572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790700068255611-789-0/minicluster-data/master-0/wal/instance:
uuid: "ddf5cbc0146144f0b8e0519d3eab394a"
format_stamp: "Formatted at 2026-09-29 16:44:54 on dist-test-slave-l31f"
server_key: "fd524beb81fc214658f7d313773c83da"
server_key_iv: "372f716309d798e7ca6acd7d5872bd88"
server_key_version: "encryptionkey@0"
I20260929 16:44:54.749544 14572 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 16:44:54.753844 14587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:44:54.755709 14572 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:44:54.755944 14572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "ddf5cbc0146144f0b8e0519d3eab394a"
format_stamp: "Formatted at 2026-09-29 16:44:54 on dist-test-slave-l31f"
server_key: "fd524beb81fc214658f7d313773c83da"
server_key_iv: "372f716309d798e7ca6acd7d5872bd88"
server_key_version: "encryptionkey@0"
I20260929 16:44:54.756196 14572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:44:54.769207 14572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:44:54.774780 14572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:44:54.775208 14572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:44:54.795346 14572 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.126:44385
I20260929 16:44:54.795352 14638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.126:44385 every 8 connection(s)
I20260929 16:44:54.797411 14572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:44:54.801940 14639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:44:54.804991 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 14572
I20260929 16:44:54.805255 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790700068255611-789-0/minicluster-data/master-0/wal/instance
I20260929 16:44:54.806007 789 external_mini_cluster.cc:1428] Setting key d77861c1abd60b6c72ddf9395d16a9f0
I20260929 16:44:54.815692 14639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddf5cbc0146144f0b8e0519d3eab394a: Bootstrap starting.
I20260929 16:44:54.820590 14639 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ddf5cbc0146144f0b8e0519d3eab394a: Neither blocks nor log segments found. Creating new log.
I20260929 16:44:54.821837 14639 log.cc:824] T 00000000000000000000000000000000 P ddf5cbc0146144f0b8e0519d3eab394a: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:44:54 dist-test-slave-l31f krb5kdc[14559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:44:54.825430 14639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddf5cbc0146144f0b8e0519d3eab394a: No bootstrap required, opened a new log
I20260929 16:44:54.833084 14642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:54.808365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59260 (local address 127.0.197.126:44385)
0929 16:44:54.809394 (+ 1029us) server_negotiation.cc:206] Beginning negotiation
0929 16:44:54.809409 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:44:54.809466 (+ 57us) server_negotiation.cc:407] Connection header received
0929 16:44:54.811580 (+ 2114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:44:54.811609 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:44:54.812115 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:44:54.812636 (+ 521us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:44:54.814474 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:54.816156 (+ 1682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:54.818291 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:54.819280 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:54.819558 (+ 278us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:54.824396 (+ 4838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:44:54.824406 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:44:54.824425 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:44:54.824466 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:44:54.828053 (+ 3587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:54.828979 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:54.828987 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:54.828995 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:54.829203 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:54.829664 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:54.829675 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:54.829678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:54.830351 (+ 673us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:44:54.830730 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0929 16:44:54.831259 (+ 529us) server_negotiation.cc:299] Negotiation successful
0929 16:44:54.831803 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":776,"thread_start_us":519,"threads_started":1}
I20260929 16:44:54.835403 14639 raft_consensus.cc:348] T 00000000000000000000000000000000 P ddf5cbc0146144f0b8e0519d3eab394a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf5cbc0146144f0b8e0519d3eab394a" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 44385 } }
I20260929 16:44:54.835876 14639 raft_consensus.cc:374] T 00000000000000000000000000000000 P ddf5cbc0146144f0b8e0519d3eab394a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:44:54.836009 14639 raft_consensus.cc:729] T 00000000000000000000000000000000 P ddf5cbc0146144f0b8e0519d3eab394a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddf5cbc0146144f0b8e0519d3eab394a, State: Initialized, Role: FOLLOWER
I20260929 16:44:54.836750 14639 consensus_queue.cc:260] T 00000000000000000000000000000000 P ddf5cbc0146144f0b8e0519d3eab394a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf5cbc0146144f0b8e0519d3eab394a" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 44385 } }
I20260929 16:44:54.837039 14639 raft_consensus.cc:388] T 00000000000000000000000000000000 P ddf5cbc0146144f0b8e0519d3eab394a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:44:54.837209 14639 raft_consensus.cc:482] T 00000000000000000000000000000000 P ddf5cbc0146144f0b8e0519d3eab394a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:44:54.837424 14639 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ddf5cbc0146144f0b8e0519d3eab394a [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:44:54.838780 789 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:44:54.840205 14639 raft_consensus.cc:504] T 00000000000000000000000000000000 P ddf5cbc0146144f0b8e0519d3eab394a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf5cbc0146144f0b8e0519d3eab394a" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 44385 } }
I20260929 16:44:54.840848 14639 leader_election.cc:302] T 00000000000000000000000000000000 P ddf5cbc0146144f0b8e0519d3eab394a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ddf5cbc0146144f0b8e0519d3eab394a; no voters:
I20260929 16:44:54.841657 14639 leader_election.cc:288] T 00000000000000000000000000000000 P ddf5cbc0146144f0b8e0519d3eab394a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:44:54.841797 14644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ddf5cbc0146144f0b8e0519d3eab394a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:44:54.843287 14644 raft_consensus.cc:686] T 00000000000000000000000000000000 P ddf5cbc0146144f0b8e0519d3eab394a [term 1 LEADER]: Becoming Leader. State: Replica: ddf5cbc0146144f0b8e0519d3eab394a, State: Running, Role: LEADER
I20260929 16:44:54.844074 14644 consensus_queue.cc:237] T 00000000000000000000000000000000 P ddf5cbc0146144f0b8e0519d3eab394a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf5cbc0146144f0b8e0519d3eab394a" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 44385 } }
I20260929 16:44:54.844631 14639 sys_catalog.cc:564] T 00000000000000000000000000000000 P ddf5cbc0146144f0b8e0519d3eab394a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:44:54.847957 14645 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddf5cbc0146144f0b8e0519d3eab394a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ddf5cbc0146144f0b8e0519d3eab394a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf5cbc0146144f0b8e0519d3eab394a" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 44385 } } }
I20260929 16:44:54.849879 14645 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddf5cbc0146144f0b8e0519d3eab394a [sys.catalog]: This master's current role is: LEADER
I20260929 16:44:54.847854 14646 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddf5cbc0146144f0b8e0519d3eab394a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ddf5cbc0146144f0b8e0519d3eab394a. Latest consensus state: current_term: 1 leader_uuid: "ddf5cbc0146144f0b8e0519d3eab394a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddf5cbc0146144f0b8e0519d3eab394a" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 44385 } } }
I20260929 16:44:54.850401 14646 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddf5cbc0146144f0b8e0519d3eab394a [sys.catalog]: This master's current role is: LEADER
I20260929 16:44:54.860399 14657 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:44:54.865811 14657 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:44:54.875145 14657 catalog_manager.cc:1347] Generated new cluster ID: ac56dcf1373e46f68b21996db0e1018a
I20260929 16:44:54.875334 14657 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:44:54.890949 14657 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:44:54.892650 14657 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:44:54.900101 14657 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ddf5cbc0146144f0b8e0519d3eab394a: Generated new TSK 0
I20260929 16:44:54.901134 14657 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:44:54.956498 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 14572
I20260929 16:44:54.957870 14634 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:44:55.061173 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 14572
2026-09-29T16:44:55Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2114 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:44:55 dist-test-slave-l31f krb5kdc[14667](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:44:55 dist-test-slave-l31f krb5kdc[14667](info): set up 2 sockets
Sep 29 16:44:55 dist-test-slave-l31f krb5kdc[14667](info): commencing operation
krb5kdc: starting...
W20260929 16:44:56.204176 789 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/test-user.keytab.
Sep 29 16:44:56 dist-test-slave-l31f krb5kdc[14667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:44:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:44:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.197.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
I20260929 16:44:56.467655 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:36941
--webserver_interface=127.0.197.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:36245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.126:36941
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:44:56.840739 14680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:44:56.841085 14680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:44:56.841193 14680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:44:56.849943 14680 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:44:56.850093 14680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:44:56.850150 14680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:44:56.850206 14680 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:44:56.850253 14680 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:44:56.861193 14680 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:36245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.197.126:36941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:36941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.197.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:44:56.862622 14680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:44:56.864084 14680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:44:56.877142 14686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:44:56.877257 14685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:44:56.877712 14688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:44:56.878012 14680 server_base.cc:1034] running on GCE node
I20260929 16:44:56.878928 14680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:44:56.880761 14680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:44:56.882071 14680 hybrid_clock.cc:648] HybridClock initialized: now 1790700296882024 us; error 49 us; skew 500 ppm
I20260929 16:44:56.882515 14680 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:44:56 dist-test-slave-l31f krb5kdc[14667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700296, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:44:56.885912 14680 init.cc:377] Logged in from keytab as kudu/127.0.197.126@KRBTEST.COM (short username kudu)
I20260929 16:44:56.887723 14680 webserver.cc:460] Webserver started at http://127.0.197.126:44179/ using document root <none> and password file <none>
I20260929 16:44:56.888446 14680 fs_manager.cc:359] Metadata directory not provided
I20260929 16:44:56.888564 14680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:44:56.888818 14680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:44:56.892295 14680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/master-0/data/instance:
uuid: "54cae7cc4b3b4fd5adff223d09b60644"
format_stamp: "Formatted at 2026-09-29 16:44:56 on dist-test-slave-l31f"
server_key: "dc88f0acbb71b42de1e581a465a44380"
server_key_iv: "03d41cedf079e423cfc4bc7198ee5f6d"
server_key_version: "encryptionkey@0"
I20260929 16:44:56.893143 14680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/master-0/wal/instance:
uuid: "54cae7cc4b3b4fd5adff223d09b60644"
format_stamp: "Formatted at 2026-09-29 16:44:56 on dist-test-slave-l31f"
server_key: "dc88f0acbb71b42de1e581a465a44380"
server_key_iv: "03d41cedf079e423cfc4bc7198ee5f6d"
server_key_version: "encryptionkey@0"
I20260929 16:44:56.899374 14680 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 16:44:56.904014 14695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:44:56.905821 14680 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:44:56.906132 14680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "54cae7cc4b3b4fd5adff223d09b60644"
format_stamp: "Formatted at 2026-09-29 16:44:56 on dist-test-slave-l31f"
server_key: "dc88f0acbb71b42de1e581a465a44380"
server_key_iv: "03d41cedf079e423cfc4bc7198ee5f6d"
server_key_version: "encryptionkey@0"
I20260929 16:44:56.906353 14680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:44:56.925266 14680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:44:56.930063 14680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:44:56.930418 14680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:44:56.951814 14746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.126:36941 every 8 connection(s)
I20260929 16:44:56.951906 14680 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.126:36941
I20260929 16:44:56.953727 14680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:44:56.957976 14747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:44:56.963217 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 14680
I20260929 16:44:56.963457 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/master-0/wal/instance
I20260929 16:44:56.964124 789 external_mini_cluster.cc:1428] Setting key f6a2da86915b9e07cbcfab8e4f8e69aa
I20260929 16:44:56.970327 14747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644: Bootstrap starting.
I20260929 16:44:56.974457 14747 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644: Neither blocks nor log segments found. Creating new log.
I20260929 16:44:56.975896 14747 log.cc:824] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644: Log is configured to *not* fsync() on all Append() calls
I20260929 16:44:56.979202 14747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644: No bootstrap required, opened a new log
Sep 29 16:44:56 dist-test-slave-l31f krb5kdc[14667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:44:56.987999 14747 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54cae7cc4b3b4fd5adff223d09b60644" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 36941 } }
I20260929 16:44:56.988580 14747 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:44:56.988716 14747 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54cae7cc4b3b4fd5adff223d09b60644, State: Initialized, Role: FOLLOWER
I20260929 16:44:56.989460 14747 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54cae7cc4b3b4fd5adff223d09b60644" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 36941 } }
I20260929 16:44:56.989737 14747 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:44:56.989780 14750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:56.966372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53806 (local address 127.0.197.126:36941)
0929 16:44:56.967356 (+ 984us) server_negotiation.cc:206] Beginning negotiation
0929 16:44:56.967369 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:44:56.967419 (+ 50us) server_negotiation.cc:407] Connection header received
0929 16:44:56.969227 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:44:56.969253 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:44:56.969700 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:44:56.970078 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:44:56.971983 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:56.973372 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:56.975429 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:56.976309 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:56.976543 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:56.980909 (+ 4366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:44:56.980918 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:44:56.980937 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:44:56.980982 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:44:56.984642 (+ 3660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:56.985612 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:56.985620 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:56.985628 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:56.985829 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:56.986315 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:56.986319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:56.986321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:56.987081 (+ 760us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:44:56.987430 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0929 16:44:56.987932 (+ 502us) server_negotiation.cc:299] Negotiation successful
0929 16:44:56.988486 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":722,"thread_start_us":445,"threads_started":1}
I20260929 16:44:56.989920 14747 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:44:56.990130 14747 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:44:56.992777 14747 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54cae7cc4b3b4fd5adff223d09b60644" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 36941 } }
I20260929 16:44:56.993462 14747 leader_election.cc:302] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54cae7cc4b3b4fd5adff223d09b60644; no voters:
I20260929 16:44:56.994062 14747 leader_election.cc:288] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:44:56.994251 14752 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:44:57.000662 14747 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:44:57.001003 14752 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644 [term 1 LEADER]: Becoming Leader. State: Replica: 54cae7cc4b3b4fd5adff223d09b60644, State: Running, Role: LEADER
I20260929 16:44:57.001817 14752 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54cae7cc4b3b4fd5adff223d09b60644" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 36941 } }
I20260929 16:44:57.008510 14753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54cae7cc4b3b4fd5adff223d09b60644" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54cae7cc4b3b4fd5adff223d09b60644" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 36941 } } }
I20260929 16:44:57.009277 14753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644 [sys.catalog]: This master's current role is: LEADER
I20260929 16:44:57.009084 14754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54cae7cc4b3b4fd5adff223d09b60644. Latest consensus state: current_term: 1 leader_uuid: "54cae7cc4b3b4fd5adff223d09b60644" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54cae7cc4b3b4fd5adff223d09b60644" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 36941 } } }
I20260929 16:44:57.010366 14754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644 [sys.catalog]: This master's current role is: LEADER
W20260929 16:44:57.013962 14766 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:44:57.014154 14766 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:44:57.014297 14767 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:44:57.016146 14767 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:44:57.025709 14767 catalog_manager.cc:1347] Generated new cluster ID: 6abb2c0bcc98481f9103fab29127bfbd
I20260929 16:44:57.025856 14767 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:44:57.040939 14767 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:44:57.042918 14767 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:44:57.070403 14767 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644: Generated new TSK 0
I20260929 16:44:57.071609 14767 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.197.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab.
Entry for principal kudu/127.0.197.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab.
I20260929 16:44:57.131978 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.65:0
--local_ip_for_outbound_sockets=127.0.197.65
--webserver_interface=127.0.197.65
--webserver_port=0
--tserver_master_addrs=127.0.197.126:36941
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab
--principal=kudu/127.0.197.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:36245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:44:57.496263 14773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:44:57.496583 14773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:44:57.496683 14773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:44:57.504772 14773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:44:57.504922 14773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:44:57.505051 14773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.65
I20260929 16:44:57.515905 14773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:36245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab
--principal=kudu/127.0.197.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.197.65
--webserver_port=0
--tserver_master_addrs=127.0.197.126:36941
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.65
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:44:57.517334 14773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:44:57.518707 14773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:44:57.533561 14778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:44:57.533617 14781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:44:57.533672 14779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:44:57.534423 14773 server_base.cc:1034] running on GCE node
I20260929 16:44:57.535220 14773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:44:57.536330 14773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:44:57.537585 14773 hybrid_clock.cc:648] HybridClock initialized: now 1790700297537510 us; error 69 us; skew 500 ppm
I20260929 16:44:57.537941 14773 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:44:57 dist-test-slave-l31f krb5kdc[14667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:44:57.541657 14773 init.cc:377] Logged in from keytab as kudu/127.0.197.65@KRBTEST.COM (short username kudu)
I20260929 16:44:57.543512 14773 webserver.cc:460] Webserver started at http://127.0.197.65:46237/ using document root <none> and password file <none>
I20260929 16:44:57.544303 14773 fs_manager.cc:359] Metadata directory not provided
I20260929 16:44:57.544468 14773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:44:57.544804 14773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:44:57.548087 14773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-0/data/instance:
uuid: "4701b5d2213f4db18e70cc60aa1298bd"
format_stamp: "Formatted at 2026-09-29 16:44:57 on dist-test-slave-l31f"
server_key: "8e4e3938bdcc38fa9f103ea5f3bc03ab"
server_key_iv: "65c157d9d03ce08f0fa7590964447678"
server_key_version: "encryptionkey@0"
I20260929 16:44:57.548995 14773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-0/wal/instance:
uuid: "4701b5d2213f4db18e70cc60aa1298bd"
format_stamp: "Formatted at 2026-09-29 16:44:57 on dist-test-slave-l31f"
server_key: "8e4e3938bdcc38fa9f103ea5f3bc03ab"
server_key_iv: "65c157d9d03ce08f0fa7590964447678"
server_key_version: "encryptionkey@0"
I20260929 16:44:57.555611 14773 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 16:44:57.559860 14788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:44:57.561589 14773 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:44:57.561826 14773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-0/wal
uuid: "4701b5d2213f4db18e70cc60aa1298bd"
format_stamp: "Formatted at 2026-09-29 16:44:57 on dist-test-slave-l31f"
server_key: "8e4e3938bdcc38fa9f103ea5f3bc03ab"
server_key_iv: "65c157d9d03ce08f0fa7590964447678"
server_key_version: "encryptionkey@0"
I20260929 16:44:57.562083 14773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:44:57.584335 14773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:44:57.589155 14773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:44:57.589497 14773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:44:57.590571 14773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:44:57.592142 14773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:44:57.592264 14773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:44:57.592417 14773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:44:57.592494 14773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:44:57.626868 14773 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.65:43425
I20260929 16:44:57.626921 14900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.65:43425 every 8 connection(s)
I20260929 16:44:57.628297 14773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
I20260929 16:44:57.638834 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 14773
I20260929 16:44:57.639138 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-0/wal/instance
I20260929 16:44:57.640055 789 external_mini_cluster.cc:1428] Setting key a464131297e612d0b53a148fd9962981
Sep 29 16:44:57 dist-test-slave-l31f krb5kdc[14667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.65@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:44:57.653354 14903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:57.631609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:52119 (local address 127.0.197.126:36941)
0929 16:44:57.632124 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0929 16:44:57.632129 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:44:57.633220 (+ 1091us) server_negotiation.cc:407] Connection header received
0929 16:44:57.635360 (+ 2140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:44:57.635367 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:44:57.635476 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:44:57.635622 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:44:57.637356 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:57.638173 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:57.641134 (+ 2961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:57.641653 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:57.641769 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:57.646539 (+ 4770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:44:57.646545 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:44:57.646549 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:44:57.646587 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:44:57.649469 (+ 2882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:57.650362 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:57.650367 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:57.650369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:57.650500 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:57.651115 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:57.651125 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:57.651131 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:57.651618 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:44:57.651780 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 16:44:57.652830 (+ 1050us) server_negotiation.cc:299] Negotiation successful
0929 16:44:57.653053 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":404,"thread_start_us":271,"threads_started":1}
I20260929 16:44:57.654863 14904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:57.632229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.65:52119)
0929 16:44:57.633002 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0929 16:44:57.633078 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0929 16:44:57.634974 (+ 1896us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:44:57.635902 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:44:57.635911 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:44:57.636506 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:44:57.637146 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:44:57.637166 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:44:57.638449 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:44:57.638457 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:44:57.640908 (+ 2451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:44:57.640926 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:44:57.641930 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:44:57.641939 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:44:57.642297 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:57.643104 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:44:57.643145 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:44:57.646243 (+ 3098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:44:57.649667 (+ 3424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:44:57.649678 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:44:57.649707 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:44:57.650204 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:44:57.650625 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:44:57.650631 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:44:57.650635 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:44:57.650904 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:44:57.651796 (+ 892us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:44:57.651806 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:44:57.652328 (+ 522us) client_negotiation.cc:769] Sending connection context
0929 16:44:57.652671 (+ 343us) client_negotiation.cc:240] Negotiation successful
0929 16:44:57.653300 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":304,"threads_started":1}
I20260929 16:44:57.656771 14901 heartbeater.cc:347] Connected to a master server at 127.0.197.126:36941
I20260929 16:44:57.657131 14901 heartbeater.cc:464] Registering TS with master...
I20260929 16:44:57.657943 14901 heartbeater.cc:511] Master 127.0.197.126:36941 requested a full tablet report, sending...
I20260929 16:44:57.660490 14712 ts_manager.cc:194] Registered new tserver with Master: 4701b5d2213f4db18e70cc60aa1298bd (127.0.197.65:43425)
I20260929 16:44:57.662338 14712 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.65@KRBTEST.COM'} at 127.0.197.65:52119
WARNING: no policy specified for kudu/127.0.197.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab.
Entry for principal kudu/127.0.197.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab.
I20260929 16:44:57.698839 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.66:0
--local_ip_for_outbound_sockets=127.0.197.66
--webserver_interface=127.0.197.66
--webserver_port=0
--tserver_master_addrs=127.0.197.126:36941
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab
--principal=kudu/127.0.197.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:36245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:44:58.056730 14907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:44:58.057125 14907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:44:58.057307 14907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:44:58.065076 14907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:44:58.065264 14907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:44:58.065464 14907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.66
I20260929 16:44:58.076715 14907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:36245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab
--principal=kudu/127.0.197.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.197.66
--webserver_port=0
--tserver_master_addrs=127.0.197.126:36941
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.66
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:44:58.078346 14907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:44:58.079900 14907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:44:58.094626 14912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:44:58.094659 14915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:44:58.094605 14913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:44:58.095716 14907 server_base.cc:1034] running on GCE node
I20260929 16:44:58.096423 14907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:44:58.097564 14907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:44:58.098836 14907 hybrid_clock.cc:648] HybridClock initialized: now 1790700298098785 us; error 81 us; skew 500 ppm
I20260929 16:44:58.099305 14907 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:44:58 dist-test-slave-l31f krb5kdc[14667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:44:58.109589 14907 init.cc:377] Logged in from keytab as kudu/127.0.197.66@KRBTEST.COM (short username kudu)
I20260929 16:44:58.111721 14907 webserver.cc:460] Webserver started at http://127.0.197.66:37485/ using document root <none> and password file <none>
I20260929 16:44:58.112588 14907 fs_manager.cc:359] Metadata directory not provided
I20260929 16:44:58.112769 14907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:44:58.113116 14907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:44:58.116379 14907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-1/data/instance:
uuid: "023b4695426f4b49a8236ecbc75651b0"
format_stamp: "Formatted at 2026-09-29 16:44:58 on dist-test-slave-l31f"
server_key: "1a95731ce82dbc3a942b69b2f0af378e"
server_key_iv: "edd89e14647e7f8283a0c390d16d397f"
server_key_version: "encryptionkey@0"
I20260929 16:44:58.117215 14907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-1/wal/instance:
uuid: "023b4695426f4b49a8236ecbc75651b0"
format_stamp: "Formatted at 2026-09-29 16:44:58 on dist-test-slave-l31f"
server_key: "1a95731ce82dbc3a942b69b2f0af378e"
server_key_iv: "edd89e14647e7f8283a0c390d16d397f"
server_key_version: "encryptionkey@0"
I20260929 16:44:58.123513 14907 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 16:44:58.127651 14922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:44:58.129256 14907 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:44:58.129477 14907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-1/wal
uuid: "023b4695426f4b49a8236ecbc75651b0"
format_stamp: "Formatted at 2026-09-29 16:44:58 on dist-test-slave-l31f"
server_key: "1a95731ce82dbc3a942b69b2f0af378e"
server_key_iv: "edd89e14647e7f8283a0c390d16d397f"
server_key_version: "encryptionkey@0"
I20260929 16:44:58.129712 14907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:44:58.151852 14907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:44:58.157222 14907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:44:58.157583 14907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:44:58.158763 14907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:44:58.160389 14907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:44:58.160532 14907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:44:58.160706 14907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:44:58.160785 14907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:44:58.200712 14907 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.66:44037
I20260929 16:44:58.200779 15034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.66:44037 every 8 connection(s)
I20260929 16:44:58.202384 14907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
I20260929 16:44:58.205188 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 14907
I20260929 16:44:58.205410 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-1/wal/instance
I20260929 16:44:58.205942 789 external_mini_cluster.cc:1428] Setting key 30bf5936c2079610be014398da851da4
Sep 29 16:44:58 dist-test-slave-l31f krb5kdc[14667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.66@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:44:58.229430 15037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:58.205650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:38439 (local address 127.0.197.126:36941)
0929 16:44:58.206239 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0929 16:44:58.206246 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:44:58.207639 (+ 1393us) server_negotiation.cc:407] Connection header received
0929 16:44:58.209640 (+ 2001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:44:58.209650 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:44:58.209751 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:44:58.209910 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:44:58.212222 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:58.213470 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:58.215387 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:58.215893 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:58.216017 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:58.221211 (+ 5194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:44:58.221220 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:44:58.221228 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:44:58.221282 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:44:58.225374 (+ 4092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:58.226450 (+ 1076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:58.226459 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:58.226465 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:58.226660 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:58.227212 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:58.227220 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:58.227223 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:58.227713 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:44:58.227889 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 16:44:58.228776 (+ 887us) server_negotiation.cc:299] Negotiation successful
0929 16:44:58.229037 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":271,"threads_started":1}
I20260929 16:44:58.230576 15038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:58.206444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.66:38439)
0929 16:44:58.207435 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0929 16:44:58.207494 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 16:44:58.209291 (+ 1797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:44:58.210278 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:44:58.210287 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:44:58.210908 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:44:58.211970 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:44:58.211998 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:44:58.213707 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:44:58.213712 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:44:58.215197 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:44:58.215213 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:44:58.216205 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:44:58.216213 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:44:58.216587 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:58.217447 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:44:58.217479 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:44:58.220822 (+ 3343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:44:58.225631 (+ 4809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:44:58.225640 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:44:58.225659 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:44:58.226255 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:44:58.226783 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:44:58.226786 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:44:58.226789 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:44:58.226989 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:44:58.227961 (+ 972us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:44:58.227968 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:44:58.228357 (+ 389us) client_negotiation.cc:769] Sending connection context
0929 16:44:58.228669 (+ 312us) client_negotiation.cc:240] Negotiation successful
0929 16:44:58.229280 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"thread_start_us":427,"threads_started":1}
WARNING: no policy specified for kudu/127.0.197.67@KRBTEST.COM; defaulting to no policy
I20260929 16:44:58.232333 15035 heartbeater.cc:347] Connected to a master server at 127.0.197.126:36941
I20260929 16:44:58.232805 15035 heartbeater.cc:464] Registering TS with master...
I20260929 16:44:58.233675 15035 heartbeater.cc:511] Master 127.0.197.126:36941 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.67@KRBTEST.COM" created.
I20260929 16:44:58.235774 14712 ts_manager.cc:194] Registered new tserver with Master: 023b4695426f4b49a8236ecbc75651b0 (127.0.197.66:44037)
I20260929 16:44:58.236670 14712 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.66@KRBTEST.COM'} at 127.0.197.66:38439
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab.
Entry for principal kudu/127.0.197.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab.
I20260929 16:44:58.267598 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.67:0
--local_ip_for_outbound_sockets=127.0.197.67
--webserver_interface=127.0.197.67
--webserver_port=0
--tserver_master_addrs=127.0.197.126:36941
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab
--principal=kudu/127.0.197.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:36245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:44:58.618043 15041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:44:58.618366 15041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:44:58.618541 15041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:44:58.626008 15041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:44:58.626228 15041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:44:58.626403 15041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.67
I20260929 16:44:58.639183 15041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:36245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab
--principal=kudu/127.0.197.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.197.67
--webserver_port=0
--tserver_master_addrs=127.0.197.126:36941
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.67
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:44:58.640499 15041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:44:58.641803 15041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:44:58.656668 15046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:44:58.656703 15049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:44:58.656618 15047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:44:58.657909 15041 server_base.cc:1034] running on GCE node
I20260929 16:44:58.658740 15041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:44:58.659929 15041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:44:58.661242 15041 hybrid_clock.cc:648] HybridClock initialized: now 1790700298661168 us; error 70 us; skew 500 ppm
I20260929 16:44:58.661628 15041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:44:58 dist-test-slave-l31f krb5kdc[14667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:44:58.665294 15041 init.cc:377] Logged in from keytab as kudu/127.0.197.67@KRBTEST.COM (short username kudu)
I20260929 16:44:58.666283 14901 heartbeater.cc:503] Master 127.0.197.126:36941 was elected leader, sending a full tablet report...
I20260929 16:44:58.667459 15041 webserver.cc:460] Webserver started at http://127.0.197.67:42641/ using document root <none> and password file <none>
I20260929 16:44:58.668321 15041 fs_manager.cc:359] Metadata directory not provided
I20260929 16:44:58.668480 15041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:44:58.668823 15041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:44:58.672063 15041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-2/data/instance:
uuid: "5553740df9ab445ba3dc8abe4d5b8e58"
format_stamp: "Formatted at 2026-09-29 16:44:58 on dist-test-slave-l31f"
server_key: "5f32c8e03343b339f2d659274da4b283"
server_key_iv: "070c2c94dff4867ddfe64e37252b4dc9"
server_key_version: "encryptionkey@0"
I20260929 16:44:58.672972 15041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-2/wal/instance:
uuid: "5553740df9ab445ba3dc8abe4d5b8e58"
format_stamp: "Formatted at 2026-09-29 16:44:58 on dist-test-slave-l31f"
server_key: "5f32c8e03343b339f2d659274da4b283"
server_key_iv: "070c2c94dff4867ddfe64e37252b4dc9"
server_key_version: "encryptionkey@0"
I20260929 16:44:58.678648 15041 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 16:44:58.682518 15056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:44:58.684257 15041 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:44:58.684475 15041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-2/wal
uuid: "5553740df9ab445ba3dc8abe4d5b8e58"
format_stamp: "Formatted at 2026-09-29 16:44:58 on dist-test-slave-l31f"
server_key: "5f32c8e03343b339f2d659274da4b283"
server_key_iv: "070c2c94dff4867ddfe64e37252b4dc9"
server_key_version: "encryptionkey@0"
I20260929 16:44:58.684724 15041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:44:58.716487 15041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:44:58.721267 15041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:44:58.721602 15041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:44:58.722905 15041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:44:58.724440 15041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:44:58.724558 15041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:44:58.724715 15041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:44:58.724797 15041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:44:58.759500 15041 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.67:33421
I20260929 16:44:58.759549 15168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.67:33421 every 8 connection(s)
I20260929 16:44:58.760967 15041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
I20260929 16:44:58.763062 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 15041
I20260929 16:44:58.763268 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-2/wal/instance
I20260929 16:44:58.763746 789 external_mini_cluster.cc:1428] Setting key 7518e2ca19699913d8fc730d678e98a9
Sep 29 16:44:58 dist-test-slave-l31f krb5kdc[14667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700298, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.67@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:44:58.784387 15171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:58.764615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:38407 (local address 127.0.197.126:36941)
0929 16:44:58.765170 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0929 16:44:58.765177 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:44:58.766554 (+ 1377us) server_negotiation.cc:407] Connection header received
0929 16:44:58.768717 (+ 2163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:44:58.768722 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:44:58.768796 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:44:58.768899 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:44:58.770543 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:58.771442 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:58.773105 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:58.773533 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:58.773656 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:58.777546 (+ 3890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:44:58.777552 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:44:58.777556 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:44:58.777594 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:44:58.780877 (+ 3283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:58.781670 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:58.781673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:58.781676 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:58.781795 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:58.782318 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:58.782321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:58.782323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:58.782616 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:44:58.782730 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 16:44:58.783830 (+ 1100us) server_negotiation.cc:299] Negotiation successful
0929 16:44:58.784087 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":235,"threads_started":1}
I20260929 16:44:58.785199 15172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:58.765372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.67:38407)
0929 16:44:58.766377 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0929 16:44:58.766432 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 16:44:58.768385 (+ 1953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:44:58.769132 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:44:58.769141 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:44:58.769721 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:44:58.770333 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:44:58.770352 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:44:58.771614 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:44:58.771618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:44:58.772924 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:44:58.772938 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:44:58.773790 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:44:58.773796 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:44:58.774018 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:58.774789 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:44:58.774814 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:44:58.777255 (+ 2441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:44:58.781040 (+ 3785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:44:58.781049 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:44:58.781066 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:44:58.781552 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:44:58.781933 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:44:58.781937 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:44:58.781940 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:44:58.782195 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:44:58.782744 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:44:58.782751 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:44:58.783293 (+ 542us) client_negotiation.cc:769] Sending connection context
0929 16:44:58.783558 (+ 265us) client_negotiation.cc:240] Negotiation successful
0929 16:44:58.784021 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":448,"threads_started":1}
I20260929 16:44:58.786883 15169 heartbeater.cc:347] Connected to a master server at 127.0.197.126:36941
I20260929 16:44:58.787277 15169 heartbeater.cc:464] Registering TS with master...
I20260929 16:44:58.788092 15169 heartbeater.cc:511] Master 127.0.197.126:36941 requested a full tablet report, sending...
I20260929 16:44:58.789860 14712 ts_manager.cc:194] Registered new tserver with Master: 5553740df9ab445ba3dc8abe4d5b8e58 (127.0.197.67:33421)
I20260929 16:44:58.790812 14712 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.67@KRBTEST.COM'} at 127.0.197.67:38407
I20260929 16:44:58.792990 789 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:44:58.812364 15171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:58.797821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53808 (local address 127.0.197.126:36941)
0929 16:44:58.798000 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 16:44:58.798005 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:44:58.798497 (+ 492us) server_negotiation.cc:407] Connection header received
0929 16:44:58.798635 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:44:58.798639 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:44:58.798723 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:44:58.798820 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:44:58.800349 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:58.801232 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:58.802771 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:58.803307 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:58.803419 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:58.806118 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:44:58.806125 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:44:58.806128 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:44:58.806171 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:44:58.809093 (+ 2922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:58.810014 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:58.810025 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:58.810028 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:58.810233 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:58.810748 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:58.810753 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:58.810755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:58.811273 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:44:58.811478 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 16:44:58.811874 (+ 396us) server_negotiation.cc:299] Negotiation successful
0929 16:44:58.812111 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 16:44:58.817390 14712 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53808:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:44:58.824795 14712 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:44:58.859244 15184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:58.838720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.66:44037 (local address 127.0.0.1:47388)
0929 16:44:58.839875 (+ 1155us) negotiation.cc:107] Waiting for socket to connect
0929 16:44:58.839922 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 16:44:58.840111 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:44:58.848255 (+ 8144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:44:58.848263 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:44:58.848304 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:44:58.848737 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:44:58.848759 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:44:58.852851 (+ 4092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:44:58.852858 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:44:58.855861 (+ 3003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:44:58.855880 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:44:58.858008 (+ 2128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:44:58.858014 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:44:58.858247 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:58.858313 (+ 66us) client_negotiation.cc:769] Sending connection context
0929 16:44:58.858485 (+ 172us) client_negotiation.cc:240] Negotiation successful
0929 16:44:58.858739 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":886,"mutex_wait_us":6,"thread_start_us":8540,"threads_started":1}
I20260929 16:44:58.859438 15182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:58.838724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47388 (local address 127.0.197.66:44037)
0929 16:44:58.847461 (+ 8737us) server_negotiation.cc:206] Beginning negotiation
0929 16:44:58.847470 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:44:58.847505 (+ 35us) server_negotiation.cc:407] Connection header received
0929 16:44:58.847638 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:44:58.847652 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:44:58.847971 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:44:58.848254 (+ 283us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:44:58.850812 (+ 2558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:58.852616 (+ 1804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:58.856076 (+ 3460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:58.857719 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:58.857913 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:58.858145 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0929 16:44:58.858807 (+ 662us) server_negotiation.cc:299] Negotiation successful
0929 16:44:58.859089 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8576,"thread_start_us":8453,"threads_started":1}
I20260929 16:44:58.859710 15185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:58.839500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47002 (local address 127.0.197.67:33421)
0929 16:44:58.847086 (+ 7586us) server_negotiation.cc:206] Beginning negotiation
0929 16:44:58.847098 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:44:58.847725 (+ 627us) server_negotiation.cc:407] Connection header received
0929 16:44:58.848112 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:44:58.848122 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:44:58.848379 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:44:58.848623 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:44:58.851188 (+ 2565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:58.852927 (+ 1739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:58.855863 (+ 2936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:58.857702 (+ 1839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:58.857878 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:58.858180 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0929 16:44:58.859134 (+ 954us) server_negotiation.cc:299] Negotiation successful
0929 16:44:58.859345 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7354,"thread_start_us":7185,"threads_started":1}
I20260929 16:44:58.861173 15186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:58.848202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43968 (local address 127.0.197.65:43425)
0929 16:44:58.852030 (+ 3828us) server_negotiation.cc:206] Beginning negotiation
0929 16:44:58.852041 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:44:58.852570 (+ 529us) server_negotiation.cc:407] Connection header received
0929 16:44:58.852770 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:44:58.852780 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:44:58.853087 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:44:58.853337 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:44:58.854379 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:58.856014 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:58.857855 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:58.859445 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:58.859594 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:58.859826 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0929 16:44:58.860672 (+ 846us) server_negotiation.cc:299] Negotiation successful
0929 16:44:58.860846 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3622,"thread_start_us":3467,"threads_started":1}
I20260929 16:44:58.861793 15187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:58.848512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:43425 (local address 127.0.0.1:43968)
0929 16:44:58.852411 (+ 3899us) negotiation.cc:107] Waiting for socket to connect
0929 16:44:58.852433 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:44:58.852591 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:44:58.853340 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:44:58.853345 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:44:58.853380 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:44:58.854002 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:44:58.854017 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:44:58.856216 (+ 2199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:44:58.856223 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:44:58.857635 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:44:58.857651 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:44:58.859980 (+ 2329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:44:58.859988 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:44:58.860178 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:58.860220 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 16:44:58.860314 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 16:44:58.860460 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3742,"mutex_wait_us":44,"thread_start_us":3596,"threads_started":1}
I20260929 16:44:58.861817 15183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:58.838983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.67:33421 (local address 127.0.0.1:47002)
0929 16:44:58.847603 (+ 8620us) negotiation.cc:107] Waiting for socket to connect
0929 16:44:58.847624 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:44:58.847846 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:44:58.849055 (+ 1209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:44:58.849063 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:44:58.849104 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:44:58.850844 (+ 1740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:44:58.850873 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:44:58.853089 (+ 2216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:44:58.853095 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:44:58.855646 (+ 2551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:44:58.855662 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:44:58.858321 (+ 2659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:44:58.858327 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:44:58.858843 (+ 516us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:58.858888 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 16:44:58.860289 (+ 1401us) client_negotiation.cc:240] Negotiation successful
0929 16:44:58.860465 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8469,"mutex_wait_us":4,"thread_start_us":636,"threads_started":1}
I20260929 16:44:58.864168 14970 tablet_service.cc:1467] Processing CreateTablet for tablet d95ca92f213a462f98fda84f3ec9e445 (DEFAULT_TABLE table=test-table [id=6c76c50775914a54a6707fa26aab8276]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:44:58.865470 14970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d95ca92f213a462f98fda84f3ec9e445. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:44:58.865829 14836 tablet_service.cc:1467] Processing CreateTablet for tablet d95ca92f213a462f98fda84f3ec9e445 (DEFAULT_TABLE table=test-table [id=6c76c50775914a54a6707fa26aab8276]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:44:58.865829 15104 tablet_service.cc:1467] Processing CreateTablet for tablet d95ca92f213a462f98fda84f3ec9e445 (DEFAULT_TABLE table=test-table [id=6c76c50775914a54a6707fa26aab8276]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:44:58.867434 14836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d95ca92f213a462f98fda84f3ec9e445. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:44:58.867425 15104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d95ca92f213a462f98fda84f3ec9e445. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:44:58.878719 15188 tablet_bootstrap.cc:492] T d95ca92f213a462f98fda84f3ec9e445 P 023b4695426f4b49a8236ecbc75651b0: Bootstrap starting.
I20260929 16:44:58.882485 15188 tablet_bootstrap.cc:654] T d95ca92f213a462f98fda84f3ec9e445 P 023b4695426f4b49a8236ecbc75651b0: Neither blocks nor log segments found. Creating new log.
I20260929 16:44:58.883376 15189 tablet_bootstrap.cc:492] T d95ca92f213a462f98fda84f3ec9e445 P 4701b5d2213f4db18e70cc60aa1298bd: Bootstrap starting.
I20260929 16:44:58.884924 15188 log.cc:824] T d95ca92f213a462f98fda84f3ec9e445 P 023b4695426f4b49a8236ecbc75651b0: Log is configured to *not* fsync() on all Append() calls
I20260929 16:44:58.887549 15189 tablet_bootstrap.cc:654] T d95ca92f213a462f98fda84f3ec9e445 P 4701b5d2213f4db18e70cc60aa1298bd: Neither blocks nor log segments found. Creating new log.
I20260929 16:44:58.889074 15189 log.cc:824] T d95ca92f213a462f98fda84f3ec9e445 P 4701b5d2213f4db18e70cc60aa1298bd: Log is configured to *not* fsync() on all Append() calls
I20260929 16:44:58.889472 15188 tablet_bootstrap.cc:492] T d95ca92f213a462f98fda84f3ec9e445 P 023b4695426f4b49a8236ecbc75651b0: No bootstrap required, opened a new log
I20260929 16:44:58.889988 15190 tablet_bootstrap.cc:492] T d95ca92f213a462f98fda84f3ec9e445 P 5553740df9ab445ba3dc8abe4d5b8e58: Bootstrap starting.
I20260929 16:44:58.890188 15188 ts_tablet_manager.cc:1397] T d95ca92f213a462f98fda84f3ec9e445 P 023b4695426f4b49a8236ecbc75651b0: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260929 16:44:58.892836 15189 tablet_bootstrap.cc:492] T d95ca92f213a462f98fda84f3ec9e445 P 4701b5d2213f4db18e70cc60aa1298bd: No bootstrap required, opened a new log
I20260929 16:44:58.893188 15189 ts_tablet_manager.cc:1397] T d95ca92f213a462f98fda84f3ec9e445 P 4701b5d2213f4db18e70cc60aa1298bd: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260929 16:44:58.894035 15190 tablet_bootstrap.cc:654] T d95ca92f213a462f98fda84f3ec9e445 P 5553740df9ab445ba3dc8abe4d5b8e58: Neither blocks nor log segments found. Creating new log.
I20260929 16:44:58.895483 15190 log.cc:824] T d95ca92f213a462f98fda84f3ec9e445 P 5553740df9ab445ba3dc8abe4d5b8e58: Log is configured to *not* fsync() on all Append() calls
I20260929 16:44:58.902132 15189 raft_consensus.cc:348] T d95ca92f213a462f98fda84f3ec9e445 P 4701b5d2213f4db18e70cc60aa1298bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023b4695426f4b49a8236ecbc75651b0" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44037 } } peers { permanent_uuid: "4701b5d2213f4db18e70cc60aa1298bd" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43425 } } peers { permanent_uuid: "5553740df9ab445ba3dc8abe4d5b8e58" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33421 } }
I20260929 16:44:58.902544 15189 raft_consensus.cc:374] T d95ca92f213a462f98fda84f3ec9e445 P 4701b5d2213f4db18e70cc60aa1298bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:44:58.902383 15188 raft_consensus.cc:348] T d95ca92f213a462f98fda84f3ec9e445 P 023b4695426f4b49a8236ecbc75651b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023b4695426f4b49a8236ecbc75651b0" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44037 } } peers { permanent_uuid: "4701b5d2213f4db18e70cc60aa1298bd" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43425 } } peers { permanent_uuid: "5553740df9ab445ba3dc8abe4d5b8e58" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33421 } }
I20260929 16:44:58.902693 15189 raft_consensus.cc:729] T d95ca92f213a462f98fda84f3ec9e445 P 4701b5d2213f4db18e70cc60aa1298bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4701b5d2213f4db18e70cc60aa1298bd, State: Initialized, Role: FOLLOWER
I20260929 16:44:58.902844 15188 raft_consensus.cc:374] T d95ca92f213a462f98fda84f3ec9e445 P 023b4695426f4b49a8236ecbc75651b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:44:58.902976 15188 raft_consensus.cc:729] T d95ca92f213a462f98fda84f3ec9e445 P 023b4695426f4b49a8236ecbc75651b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 023b4695426f4b49a8236ecbc75651b0, State: Initialized, Role: FOLLOWER
I20260929 16:44:58.903442 15189 consensus_queue.cc:260] T d95ca92f213a462f98fda84f3ec9e445 P 4701b5d2213f4db18e70cc60aa1298bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023b4695426f4b49a8236ecbc75651b0" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44037 } } peers { permanent_uuid: "4701b5d2213f4db18e70cc60aa1298bd" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43425 } } peers { permanent_uuid: "5553740df9ab445ba3dc8abe4d5b8e58" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33421 } }
I20260929 16:44:58.903762 15190 tablet_bootstrap.cc:492] T d95ca92f213a462f98fda84f3ec9e445 P 5553740df9ab445ba3dc8abe4d5b8e58: No bootstrap required, opened a new log
I20260929 16:44:58.903702 15188 consensus_queue.cc:260] T d95ca92f213a462f98fda84f3ec9e445 P 023b4695426f4b49a8236ecbc75651b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023b4695426f4b49a8236ecbc75651b0" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44037 } } peers { permanent_uuid: "4701b5d2213f4db18e70cc60aa1298bd" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43425 } } peers { permanent_uuid: "5553740df9ab445ba3dc8abe4d5b8e58" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33421 } }
I20260929 16:44:58.904078 15190 ts_tablet_manager.cc:1397] T d95ca92f213a462f98fda84f3ec9e445 P 5553740df9ab445ba3dc8abe4d5b8e58: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.007s
I20260929 16:44:58.905393 15189 ts_tablet_manager.cc:1428] T d95ca92f213a462f98fda84f3ec9e445 P 4701b5d2213f4db18e70cc60aa1298bd: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260929 16:44:58.906528 15035 heartbeater.cc:503] Master 127.0.197.126:36941 was elected leader, sending a full tablet report...
I20260929 16:44:58.907291 15188 ts_tablet_manager.cc:1428] T d95ca92f213a462f98fda84f3ec9e445 P 023b4695426f4b49a8236ecbc75651b0: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
I20260929 16:44:58.914460 15190 raft_consensus.cc:348] T d95ca92f213a462f98fda84f3ec9e445 P 5553740df9ab445ba3dc8abe4d5b8e58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023b4695426f4b49a8236ecbc75651b0" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44037 } } peers { permanent_uuid: "4701b5d2213f4db18e70cc60aa1298bd" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43425 } } peers { permanent_uuid: "5553740df9ab445ba3dc8abe4d5b8e58" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33421 } }
I20260929 16:44:58.914947 15190 raft_consensus.cc:374] T d95ca92f213a462f98fda84f3ec9e445 P 5553740df9ab445ba3dc8abe4d5b8e58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:44:58.915100 15190 raft_consensus.cc:729] T d95ca92f213a462f98fda84f3ec9e445 P 5553740df9ab445ba3dc8abe4d5b8e58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5553740df9ab445ba3dc8abe4d5b8e58, State: Initialized, Role: FOLLOWER
I20260929 16:44:58.915725 15190 consensus_queue.cc:260] T d95ca92f213a462f98fda84f3ec9e445 P 5553740df9ab445ba3dc8abe4d5b8e58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023b4695426f4b49a8236ecbc75651b0" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44037 } } peers { permanent_uuid: "4701b5d2213f4db18e70cc60aa1298bd" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43425 } } peers { permanent_uuid: "5553740df9ab445ba3dc8abe4d5b8e58" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33421 } }
I20260929 16:44:58.917264 15169 heartbeater.cc:503] Master 127.0.197.126:36941 was elected leader, sending a full tablet report...
I20260929 16:44:58.917641 15190 ts_tablet_manager.cc:1428] T d95ca92f213a462f98fda84f3ec9e445 P 5553740df9ab445ba3dc8abe4d5b8e58: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s
W20260929 16:44:58.954958 15036 tablet.cc:2369] T d95ca92f213a462f98fda84f3ec9e445 P 023b4695426f4b49a8236ecbc75651b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:44:59.013561 15170 tablet.cc:2369] T d95ca92f213a462f98fda84f3ec9e445 P 5553740df9ab445ba3dc8abe4d5b8e58: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:44:59.023890 15195 raft_consensus.cc:482] T d95ca92f213a462f98fda84f3ec9e445 P 023b4695426f4b49a8236ecbc75651b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:44:59.024235 15195 raft_consensus.cc:504] T d95ca92f213a462f98fda84f3ec9e445 P 023b4695426f4b49a8236ecbc75651b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023b4695426f4b49a8236ecbc75651b0" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44037 } } peers { permanent_uuid: "4701b5d2213f4db18e70cc60aa1298bd" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43425 } } peers { permanent_uuid: "5553740df9ab445ba3dc8abe4d5b8e58" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33421 } }
I20260929 16:44:59.026081 15195 leader_election.cc:288] T d95ca92f213a462f98fda84f3ec9e445 P 023b4695426f4b49a8236ecbc75651b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4701b5d2213f4db18e70cc60aa1298bd (127.0.197.65:43425), 5553740df9ab445ba3dc8abe4d5b8e58 (127.0.197.67:33421)
I20260929 16:44:59.036053 15185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:59.026568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:42461 (local address 127.0.197.67:33421)
0929 16:44:59.026772 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 16:44:59.026778 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:44:59.027588 (+ 810us) server_negotiation.cc:407] Connection header received
0929 16:44:59.027780 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:44:59.027786 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:44:59.027867 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:44:59.027973 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:44:59.028767 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:59.029891 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:59.033262 (+ 3371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:59.034642 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:59.034809 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:59.034891 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 16:44:59.035621 (+ 730us) server_negotiation.cc:299] Negotiation successful
0929 16:44:59.035769 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 16:44:59.036044 15186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:59.026568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:52553 (local address 127.0.197.65:43425)
0929 16:44:59.026772 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 16:44:59.026778 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:44:59.027588 (+ 810us) server_negotiation.cc:407] Connection header received
0929 16:44:59.027780 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:44:59.027786 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:44:59.027867 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:44:59.027973 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:44:59.028767 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:59.029891 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:59.033254 (+ 3363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:59.034601 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:59.034809 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:59.034891 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 16:44:59.035621 (+ 730us) server_negotiation.cc:299] Negotiation successful
0929 16:44:59.035769 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 16:44:59.036393 15198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:59.026476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.67:33421 (local address 127.0.197.66:42461)
0929 16:44:59.027447 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0929 16:44:59.027461 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:44:59.027626 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:44:59.028021 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:44:59.028024 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:44:59.028059 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:44:59.028621 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:44:59.028628 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:44:59.030081 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:44:59.030087 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:44:59.033022 (+ 2935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:44:59.033045 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:44:59.034984 (+ 1939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:44:59.035048 (+ 64us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:44:59.035328 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:59.035425 (+ 97us) client_negotiation.cc:769] Sending connection context
0929 16:44:59.035548 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 16:44:59.035811 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":871,"mutex_wait_us":342,"spinlock_wait_cycles":3456,"thread_start_us":571,"threads_started":1}
I20260929 16:44:59.036420 15197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:59.026435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:43425 (local address 127.0.197.66:52553)
0929 16:44:59.027419 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0929 16:44:59.027444 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 16:44:59.027629 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:44:59.028009 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:44:59.028014 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:44:59.028045 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:44:59.028593 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:44:59.028613 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:44:59.030081 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:44:59.030087 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:44:59.033022 (+ 2935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:44:59.033045 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:44:59.034986 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:44:59.034993 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:44:59.035330 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:59.035425 (+ 95us) client_negotiation.cc:769] Sending connection context
0929 16:44:59.035563 (+ 138us) client_negotiation.cc:240] Negotiation successful
0929 16:44:59.035811 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":830,"mutex_wait_us":310,"spinlock_wait_cycles":47872,"thread_start_us":591,"threads_started":1}
I20260929 16:44:59.037716 15124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d95ca92f213a462f98fda84f3ec9e445" candidate_uuid: "023b4695426f4b49a8236ecbc75651b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5553740df9ab445ba3dc8abe4d5b8e58" is_pre_election: true
I20260929 16:44:59.037716 14856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d95ca92f213a462f98fda84f3ec9e445" candidate_uuid: "023b4695426f4b49a8236ecbc75651b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4701b5d2213f4db18e70cc60aa1298bd" is_pre_election: true
I20260929 16:44:59.038349 15124 raft_consensus.cc:2454] T d95ca92f213a462f98fda84f3ec9e445 P 5553740df9ab445ba3dc8abe4d5b8e58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 023b4695426f4b49a8236ecbc75651b0 in term 0.
I20260929 16:44:59.038350 14856 raft_consensus.cc:2454] T d95ca92f213a462f98fda84f3ec9e445 P 4701b5d2213f4db18e70cc60aa1298bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 023b4695426f4b49a8236ecbc75651b0 in term 0.
I20260929 16:44:59.039137 14926 leader_election.cc:302] T d95ca92f213a462f98fda84f3ec9e445 P 023b4695426f4b49a8236ecbc75651b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 023b4695426f4b49a8236ecbc75651b0, 4701b5d2213f4db18e70cc60aa1298bd; no voters:
I20260929 16:44:59.039588 15195 raft_consensus.cc:2781] T d95ca92f213a462f98fda84f3ec9e445 P 023b4695426f4b49a8236ecbc75651b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:44:59.039749 15195 raft_consensus.cc:482] T d95ca92f213a462f98fda84f3ec9e445 P 023b4695426f4b49a8236ecbc75651b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:44:59.039878 15195 raft_consensus.cc:3037] T d95ca92f213a462f98fda84f3ec9e445 P 023b4695426f4b49a8236ecbc75651b0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:44:59.042146 15195 raft_consensus.cc:504] T d95ca92f213a462f98fda84f3ec9e445 P 023b4695426f4b49a8236ecbc75651b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023b4695426f4b49a8236ecbc75651b0" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44037 } } peers { permanent_uuid: "4701b5d2213f4db18e70cc60aa1298bd" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43425 } } peers { permanent_uuid: "5553740df9ab445ba3dc8abe4d5b8e58" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33421 } }
I20260929 16:44:59.042870 15195 leader_election.cc:288] T d95ca92f213a462f98fda84f3ec9e445 P 023b4695426f4b49a8236ecbc75651b0 [CANDIDATE]: Term 1 election: Requested vote from peers 4701b5d2213f4db18e70cc60aa1298bd (127.0.197.65:43425), 5553740df9ab445ba3dc8abe4d5b8e58 (127.0.197.67:33421)
I20260929 16:44:59.043503 15124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d95ca92f213a462f98fda84f3ec9e445" candidate_uuid: "023b4695426f4b49a8236ecbc75651b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5553740df9ab445ba3dc8abe4d5b8e58"
I20260929 16:44:59.043534 14856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d95ca92f213a462f98fda84f3ec9e445" candidate_uuid: "023b4695426f4b49a8236ecbc75651b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4701b5d2213f4db18e70cc60aa1298bd"
I20260929 16:44:59.043751 15124 raft_consensus.cc:3037] T d95ca92f213a462f98fda84f3ec9e445 P 5553740df9ab445ba3dc8abe4d5b8e58 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:44:59.043766 14856 raft_consensus.cc:3037] T d95ca92f213a462f98fda84f3ec9e445 P 4701b5d2213f4db18e70cc60aa1298bd [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:44:59.046084 15124 raft_consensus.cc:2454] T d95ca92f213a462f98fda84f3ec9e445 P 5553740df9ab445ba3dc8abe4d5b8e58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 023b4695426f4b49a8236ecbc75651b0 in term 1.
I20260929 16:44:59.046097 14856 raft_consensus.cc:2454] T d95ca92f213a462f98fda84f3ec9e445 P 4701b5d2213f4db18e70cc60aa1298bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 023b4695426f4b49a8236ecbc75651b0 in term 1.
I20260929 16:44:59.046675 14926 leader_election.cc:302] T d95ca92f213a462f98fda84f3ec9e445 P 023b4695426f4b49a8236ecbc75651b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 023b4695426f4b49a8236ecbc75651b0, 4701b5d2213f4db18e70cc60aa1298bd, 5553740df9ab445ba3dc8abe4d5b8e58; no voters:
I20260929 16:44:59.047115 15195 raft_consensus.cc:2781] T d95ca92f213a462f98fda84f3ec9e445 P 023b4695426f4b49a8236ecbc75651b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:44:59.047652 15195 raft_consensus.cc:686] T d95ca92f213a462f98fda84f3ec9e445 P 023b4695426f4b49a8236ecbc75651b0 [term 1 LEADER]: Becoming Leader. State: Replica: 023b4695426f4b49a8236ecbc75651b0, State: Running, Role: LEADER
I20260929 16:44:59.048169 15195 consensus_queue.cc:237] T d95ca92f213a462f98fda84f3ec9e445 P 023b4695426f4b49a8236ecbc75651b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023b4695426f4b49a8236ecbc75651b0" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44037 } } peers { permanent_uuid: "4701b5d2213f4db18e70cc60aa1298bd" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43425 } } peers { permanent_uuid: "5553740df9ab445ba3dc8abe4d5b8e58" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33421 } }
I20260929 16:44:59.053375 14712 catalog_manager.cc:5495] T d95ca92f213a462f98fda84f3ec9e445 P 023b4695426f4b49a8236ecbc75651b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 023b4695426f4b49a8236ecbc75651b0 (127.0.197.66). New cstate: current_term: 1 leader_uuid: "023b4695426f4b49a8236ecbc75651b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "023b4695426f4b49a8236ecbc75651b0" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4701b5d2213f4db18e70cc60aa1298bd" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5553740df9ab445ba3dc8abe4d5b8e58" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33421 } health_report { overall_health: UNKNOWN } } }
I20260929 16:44:59.098627 15182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:59.092585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47392 (local address 127.0.197.66:44037)
0929 16:44:59.092845 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0929 16:44:59.092855 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:44:59.092872 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:44:59.092969 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:44:59.092972 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:44:59.093052 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:44:59.093220 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:44:59.093605 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:59.094342 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:59.095990 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:59.096419 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:59.096558 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:59.097173 (+ 615us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:44:59.097985 (+ 812us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:44:59.098145 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 16:44:59.098226 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 16:44:59.098327 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260929 16:44:59.107807 15124 raft_consensus.cc:1261] T d95ca92f213a462f98fda84f3ec9e445 P 5553740df9ab445ba3dc8abe4d5b8e58 [term 1 FOLLOWER]: Refusing update from remote peer 023b4695426f4b49a8236ecbc75651b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:44:59.107810 14856 raft_consensus.cc:1261] T d95ca92f213a462f98fda84f3ec9e445 P 4701b5d2213f4db18e70cc60aa1298bd [term 1 FOLLOWER]: Refusing update from remote peer 023b4695426f4b49a8236ecbc75651b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:44:59.108876 15199 consensus_queue.cc:1035] T d95ca92f213a462f98fda84f3ec9e445 P 023b4695426f4b49a8236ecbc75651b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5553740df9ab445ba3dc8abe4d5b8e58" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:44:59.109251 15195 consensus_queue.cc:1035] T d95ca92f213a462f98fda84f3ec9e445 P 023b4695426f4b49a8236ecbc75651b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4701b5d2213f4db18e70cc60aa1298bd" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 16:44:59.131080 14902 tablet.cc:2369] T d95ca92f213a462f98fda84f3ec9e445 P 4701b5d2213f4db18e70cc60aa1298bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:44:59.132539 15203 mvcc.cc:205] Tried to move back new op lower bound from 7334708425134272512 to 7334708424905416704. Current Snapshot: MvccSnapshot[applied={T|T < 7334708425134272512}]
I20260929 16:44:59.136348 14712 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:44:59.136793 14712 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:44:59.142378 14712 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644: Sending DeleteTablet for 3 replicas of tablet d95ca92f213a462f98fda84f3ec9e445
I20260929 16:44:59.143802 14970 tablet_service.cc:1514] Processing DeleteTablet for tablet d95ca92f213a462f98fda84f3ec9e445 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:44:59 UTC) from {username='kudu', principal='kudu/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:47388
I20260929 16:44:59.143817 14836 tablet_service.cc:1514] Processing DeleteTablet for tablet d95ca92f213a462f98fda84f3ec9e445 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:44:59 UTC) from {username='kudu', principal='kudu/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:43968
I20260929 16:44:59.144260 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 14773
I20260929 16:44:59.144381 15104 tablet_service.cc:1514] Processing DeleteTablet for tablet d95ca92f213a462f98fda84f3ec9e445 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:44:59 UTC) from {username='kudu', principal='kudu/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:47002
I20260929 16:44:59.145128 15103 tablet_service.cc:1514] Processing DeleteTablet for tablet d95ca92f213a462f98fda84f3ec9e445 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:44:59 UTC) from {username='kudu', principal='kudu/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:47002
I20260929 16:44:59.155143 15212 tablet_replica.cc:330] T d95ca92f213a462f98fda84f3ec9e445 P 4701b5d2213f4db18e70cc60aa1298bd: stopping tablet replica
I20260929 16:44:59.156076 15212 raft_consensus.cc:2229] T d95ca92f213a462f98fda84f3ec9e445 P 4701b5d2213f4db18e70cc60aa1298bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:44:59.156993 15213 tablet_replica.cc:330] T d95ca92f213a462f98fda84f3ec9e445 P 5553740df9ab445ba3dc8abe4d5b8e58: stopping tablet replica
I20260929 16:44:59.157546 15213 raft_consensus.cc:2229] T d95ca92f213a462f98fda84f3ec9e445 P 5553740df9ab445ba3dc8abe4d5b8e58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:44:59.158142 15213 raft_consensus.cc:2258] T d95ca92f213a462f98fda84f3ec9e445 P 5553740df9ab445ba3dc8abe4d5b8e58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:44:59.158806 15212 raft_consensus.cc:2258] T d95ca92f213a462f98fda84f3ec9e445 P 4701b5d2213f4db18e70cc60aa1298bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:44:59.161345 15212 ts_tablet_manager.cc:1905] T d95ca92f213a462f98fda84f3ec9e445 P 4701b5d2213f4db18e70cc60aa1298bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:44:59.161777 15213 ts_tablet_manager.cc:1905] T d95ca92f213a462f98fda84f3ec9e445 P 5553740df9ab445ba3dc8abe4d5b8e58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:44:59.169657 15212 ts_tablet_manager.cc:1918] T d95ca92f213a462f98fda84f3ec9e445 P 4701b5d2213f4db18e70cc60aa1298bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Sep 29 16:44:59 dist-test-slave-l31f krb5kdc[14667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.65@KRBTEST.COM
I20260929 16:44:59.170953 15213 ts_tablet_manager.cc:1918] T d95ca92f213a462f98fda84f3ec9e445 P 5553740df9ab445ba3dc8abe4d5b8e58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:44:59.171178 15213 log.cc:1196] T d95ca92f213a462f98fda84f3ec9e445 P 5553740df9ab445ba3dc8abe4d5b8e58: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-2/wal/wals/d95ca92f213a462f98fda84f3ec9e445
I20260929 16:44:59.170989 15211 tablet_replica.cc:330] T d95ca92f213a462f98fda84f3ec9e445 P 023b4695426f4b49a8236ecbc75651b0: stopping tablet replica
I20260929 16:44:59.171810 15213 ts_tablet_manager.cc:1939] T d95ca92f213a462f98fda84f3ec9e445 P 5553740df9ab445ba3dc8abe4d5b8e58: Deleting consensus metadata
I20260929 16:44:59.172201 15211 raft_consensus.cc:2229] T d95ca92f213a462f98fda84f3ec9e445 P 023b4695426f4b49a8236ecbc75651b0 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:44:59.173053 15211 raft_consensus.cc:2258] T d95ca92f213a462f98fda84f3ec9e445 P 023b4695426f4b49a8236ecbc75651b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:44:59.174021 14698 catalog_manager.cc:4841] TS 5553740df9ab445ba3dc8abe4d5b8e58 (127.0.197.67:33421): tablet d95ca92f213a462f98fda84f3ec9e445 (table test-table [id=6c76c50775914a54a6707fa26aab8276]) successfully deleted
W20260929 16:44:59.174737 14698 catalog_manager.cc:4808] TS 5553740df9ab445ba3dc8abe4d5b8e58 (127.0.197.67:33421): delete failed for tablet d95ca92f213a462f98fda84f3ec9e445 because the tablet was not found. No further retry: Not found: Tablet not found: d95ca92f213a462f98fda84f3ec9e445
I20260929 16:44:59.176465 15211 ts_tablet_manager.cc:1905] T d95ca92f213a462f98fda84f3ec9e445 P 023b4695426f4b49a8236ecbc75651b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:44:59.179209 15212 log.cc:1196] T d95ca92f213a462f98fda84f3ec9e445 P 4701b5d2213f4db18e70cc60aa1298bd: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-0/wal/wals/d95ca92f213a462f98fda84f3ec9e445
I20260929 16:44:59.179941 15212 ts_tablet_manager.cc:1939] T d95ca92f213a462f98fda84f3ec9e445 P 4701b5d2213f4db18e70cc60aa1298bd: Deleting consensus metadata
I20260929 16:44:59.181095 15186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:59.145606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43976 (local address 127.0.197.65:43425)
0929 16:44:59.145890 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0929 16:44:59.145899 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:44:59.157978 (+ 12079us) server_negotiation.cc:407] Connection header received
0929 16:44:59.158275 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:44:59.158282 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:44:59.158398 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:44:59.158533 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:44:59.161123 (+ 2590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:59.162295 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:59.164611 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:59.165272 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:59.165395 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:59.172569 (+ 7174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:44:59.172589 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:44:59.172608 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:44:59.172676 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:44:59.176910 (+ 4234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:59.178062 (+ 1152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:59.178085 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:59.178091 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:59.178294 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:59.178771 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:59.178777 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:59.178780 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:59.179333 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:44:59.179693 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0929 16:44:59.180332 (+ 639us) server_negotiation.cc:299] Negotiation successful
0929 16:44:59.180608 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260929 16:44:59.182699 14896 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:44:59.183177 14699 catalog_manager.cc:4841] TS 4701b5d2213f4db18e70cc60aa1298bd (127.0.197.65:43425): tablet d95ca92f213a462f98fda84f3ec9e445 (table test-table [id=6c76c50775914a54a6707fa26aab8276]) successfully deleted
I20260929 16:44:59.189621 15211 ts_tablet_manager.cc:1918] T d95ca92f213a462f98fda84f3ec9e445 P 023b4695426f4b49a8236ecbc75651b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:44:59.189858 15211 log.cc:1196] T d95ca92f213a462f98fda84f3ec9e445 P 023b4695426f4b49a8236ecbc75651b0: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-1/wal/wals/d95ca92f213a462f98fda84f3ec9e445
I20260929 16:44:59.190471 15211 ts_tablet_manager.cc:1939] T d95ca92f213a462f98fda84f3ec9e445 P 023b4695426f4b49a8236ecbc75651b0: Deleting consensus metadata
I20260929 16:44:59.192584 14699 catalog_manager.cc:4841] TS 023b4695426f4b49a8236ecbc75651b0 (127.0.197.66:44037): tablet d95ca92f213a462f98fda84f3ec9e445 (table test-table [id=6c76c50775914a54a6707fa26aab8276]) successfully deleted
I20260929 16:44:59.278465 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 14773
I20260929 16:44:59.297885 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 14907
Sep 29 16:44:59 dist-test-slave-l31f krb5kdc[14667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.66@KRBTEST.COM
I20260929 16:44:59.323208 15182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:59.298732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47404 (local address 127.0.197.66:44037)
0929 16:44:59.298910 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 16:44:59.298914 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:44:59.298933 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:44:59.299244 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:44:59.299248 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:44:59.299318 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:44:59.299438 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:44:59.301136 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:59.302188 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:59.304305 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:59.304923 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:59.305106 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:59.316530 (+ 11424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:44:59.316551 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:44:59.316569 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:44:59.316627 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:44:59.320073 (+ 3446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:59.320936 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:59.320943 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:59.320946 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:59.321097 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:59.321523 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:59.321528 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:59.321530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:59.321906 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:44:59.322282 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0929 16:44:59.322560 (+ 278us) server_negotiation.cc:299] Negotiation successful
0929 16:44:59.322805 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 16:44:59.324004 15030 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:44:59.440591 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 14907
I20260929 16:44:59.460186 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 15041
Sep 29 16:44:59 dist-test-slave-l31f krb5kdc[14667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.67@KRBTEST.COM
I20260929 16:44:59.477115 15185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:44:59.461017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47006 (local address 127.0.197.67:33421)
0929 16:44:59.461209 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 16:44:59.461215 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:44:59.461233 (+ 18us) server_negotiation.cc:407] Connection header received
0929 16:44:59.461393 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:44:59.461397 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:44:59.461466 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:44:59.461559 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:44:59.462786 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:59.463643 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:59.464899 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:44:59.465343 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:44:59.465446 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:44:59.470381 (+ 4935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:44:59.470405 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:44:59.470423 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:44:59.470475 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:44:59.473929 (+ 3454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:59.474820 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:59.474828 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:59.474831 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:59.475102 (+ 271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:44:59.475525 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:44:59.475529 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:44:59.475531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:44:59.475963 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:44:59.476311 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0929 16:44:59.476486 (+ 175us) server_negotiation.cc:299] Negotiation successful
0929 16:44:59.476688 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 16:44:59.477856 15164 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:44:59.569268 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 15041
I20260929 16:44:59.586591 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 14680
I20260929 16:44:59.587802 14742 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:44:59.700548 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 14680
I20260929 16:44:59.723644 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:36941
--webserver_interface=127.0.197.126
--webserver_port=44179
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:36245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.126:36941
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:45:00.084463 15219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:45:00.084810 15219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:45:00.084990 15219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:45:00.092672 15219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:45:00.092808 15219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:45:00.092897 15219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:45:00.093007 15219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:45:00.093097 15219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:45:00.103799 15219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:36245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.197.126:36941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:36941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.197.126
--webserver_port=44179
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:45:00.105211 15219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:45:00.106573 15219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:45:00.131883 15228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:45:00.131858 15226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:45:00.131806 15225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:45:00.132310 15219 server_base.cc:1034] running on GCE node
I20260929 16:45:00.133688 15219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:45:00.135601 15219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:45:00.136930 15219 hybrid_clock.cc:648] HybridClock initialized: now 1790700300136846 us; error 63 us; skew 500 ppm
I20260929 16:45:00.137421 15219 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:45:00 dist-test-slave-l31f krb5kdc[14667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700300, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:45:00.141403 15219 init.cc:377] Logged in from keytab as kudu/127.0.197.126@KRBTEST.COM (short username kudu)
I20260929 16:45:00.143594 15219 webserver.cc:460] Webserver started at http://127.0.197.126:44179/ using document root <none> and password file <none>
I20260929 16:45:00.144549 15219 fs_manager.cc:359] Metadata directory not provided
I20260929 16:45:00.144752 15219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:45:00.152120 15219 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260929 16:45:00.155851 15235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:45:00.157550 15219 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:45:00.157825 15219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "54cae7cc4b3b4fd5adff223d09b60644"
format_stamp: "Formatted at 2026-09-29 16:44:56 on dist-test-slave-l31f"
server_key: "dc88f0acbb71b42de1e581a465a44380"
server_key_iv: "03d41cedf079e423cfc4bc7198ee5f6d"
server_key_version: "encryptionkey@0"
I20260929 16:45:00.159124 15219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:45:00.187621 15219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:45:00.192767 15219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:45:00.193118 15219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:45:00.214697 15219 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.126:36941
I20260929 16:45:00.214720 15286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.126:36941 every 8 connection(s)
I20260929 16:45:00.216516 15219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:45:00.219645 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 15219
I20260929 16:45:00.220466 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.65:43425
--local_ip_for_outbound_sockets=127.0.197.65
--tserver_master_addrs=127.0.197.126:36941
--webserver_port=46237
--webserver_interface=127.0.197.65
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab
--principal=kudu/127.0.197.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:36245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:45:00.224623 15287 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:45:00.228396 15287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644: Bootstrap starting.
I20260929 16:45:00.249558 15287 log.cc:824] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644: Log is configured to *not* fsync() on all Append() calls
I20260929 16:45:00.265578 15287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:45:00.266350 15287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644: Bootstrap complete.
I20260929 16:45:00.276477 15287 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54cae7cc4b3b4fd5adff223d09b60644" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 36941 } }
I20260929 16:45:00.277477 15287 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54cae7cc4b3b4fd5adff223d09b60644, State: Initialized, Role: FOLLOWER
I20260929 16:45:00.278167 15287 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54cae7cc4b3b4fd5adff223d09b60644" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 36941 } }
I20260929 16:45:00.278460 15287 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:45:00.278615 15287 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:45:00.278807 15287 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644 [term 1 FOLLOWER]: Advancing to term 2
I20260929 16:45:00.283161 15287 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54cae7cc4b3b4fd5adff223d09b60644" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 36941 } }
I20260929 16:45:00.283777 15287 leader_election.cc:302] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54cae7cc4b3b4fd5adff223d09b60644; no voters:
I20260929 16:45:00.284495 15287 leader_election.cc:288] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 16:45:00.284653 15291 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 16:45:00.286238 15291 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644 [term 2 LEADER]: Becoming Leader. State: Replica: 54cae7cc4b3b4fd5adff223d09b60644, State: Running, Role: LEADER
I20260929 16:45:00.286981 15291 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54cae7cc4b3b4fd5adff223d09b60644" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 36941 } }
I20260929 16:45:00.287333 15287 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:45:00.289644 15293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54cae7cc4b3b4fd5adff223d09b60644. Latest consensus state: current_term: 2 leader_uuid: "54cae7cc4b3b4fd5adff223d09b60644" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54cae7cc4b3b4fd5adff223d09b60644" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 36941 } } }
I20260929 16:45:00.290086 15293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644 [sys.catalog]: This master's current role is: LEADER
I20260929 16:45:00.290706 15292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "54cae7cc4b3b4fd5adff223d09b60644" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54cae7cc4b3b4fd5adff223d09b60644" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 36941 } } }
I20260929 16:45:00.291033 15292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644 [sys.catalog]: This master's current role is: LEADER
I20260929 16:45:00.301397 15297 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:45:00.310957 15297 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:45:00.313263 15297 catalog_manager.cc:1259] Loaded cluster ID: 6abb2c0bcc98481f9103fab29127bfbd
I20260929 16:45:00.313489 15297 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:45:00.318570 15297 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260929 16:45:00.345185 15297 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260929 16:45:00.359481 15297 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:45:00.361606 15297 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644: Loaded TSK: 0
I20260929 16:45:00.363447 15297 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 16:45:00.608523 15289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:45:00.608857 15289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:45:00.609020 15289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:45:00.617256 15289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:45:00.617403 15289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:45:00.617604 15289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.65
I20260929 16:45:00.628402 15289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:36245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab
--principal=kudu/127.0.197.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.65:43425
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.197.65
--webserver_port=46237
--tserver_master_addrs=127.0.197.126:36941
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.65
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:45:00.629913 15289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:45:00.631417 15289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:45:00.646235 15315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:45:00.646217 15317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:45:00.646885 15314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:45:00.647374 15289 server_base.cc:1034] running on GCE node
I20260929 16:45:00.648347 15289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:45:00.650007 15289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:45:00.651598 15289 hybrid_clock.cc:648] HybridClock initialized: now 1790700300651512 us; error 95 us; skew 500 ppm
I20260929 16:45:00.652083 15289 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:45:00 dist-test-slave-l31f krb5kdc[14667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700300, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:45:00.657235 15289 init.cc:377] Logged in from keytab as kudu/127.0.197.65@KRBTEST.COM (short username kudu)
I20260929 16:45:00.659578 15289 webserver.cc:460] Webserver started at http://127.0.197.65:46237/ using document root <none> and password file <none>
I20260929 16:45:00.660737 15289 fs_manager.cc:359] Metadata directory not provided
I20260929 16:45:00.660953 15289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:45:00.669617 15289 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:45:00.673887 15324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:45:00.675765 15289 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:45:00.675987 15289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-0/wal
uuid: "4701b5d2213f4db18e70cc60aa1298bd"
format_stamp: "Formatted at 2026-09-29 16:44:57 on dist-test-slave-l31f"
server_key: "8e4e3938bdcc38fa9f103ea5f3bc03ab"
server_key_iv: "65c157d9d03ce08f0fa7590964447678"
server_key_version: "encryptionkey@0"
I20260929 16:45:00.677111 15289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:45:00.703416 15289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:45:00.708384 15289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:45:00.708705 15289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:45:00.709920 15289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:45:00.711537 15289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:45:00.711632 15289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:45:00.711776 15289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:45:00.711879 15289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:45:00.749528 15289 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.65:43425
I20260929 16:45:00.749619 15436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.65:43425 every 8 connection(s)
I20260929 16:45:00.751358 15289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
I20260929 16:45:00.756927 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 15289
I20260929 16:45:00.757581 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.66:44037
--local_ip_for_outbound_sockets=127.0.197.66
--tserver_master_addrs=127.0.197.126:36941
--webserver_port=37485
--webserver_interface=127.0.197.66
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab
--principal=kudu/127.0.197.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:36245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:45:00 dist-test-slave-l31f krb5kdc[14667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700300, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.65@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:45:00.780326 15440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:00.754868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.65:56723)
0929 16:45:00.755737 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:00.755779 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 16:45:00.757728 (+ 1949us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:45:00.759731 (+ 2003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:45:00.759739 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:45:00.760308 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:45:00.760890 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:45:00.760913 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:45:00.763672 (+ 2759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:45:00.763678 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:45:00.765241 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:45:00.765261 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:45:00.766516 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:45:00.766522 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:45:00.766726 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:00.767554 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:45:00.767579 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:45:00.769937 (+ 2358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:45:00.775266 (+ 5329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:45:00.775278 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:45:00.775295 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:45:00.775867 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:45:00.776587 (+ 720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:45:00.776593 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:45:00.776596 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:45:00.776836 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:45:00.778027 (+ 1191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:45:00.778034 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:45:00.778392 (+ 358us) client_negotiation.cc:769] Sending connection context
0929 16:45:00.778677 (+ 285us) client_negotiation.cc:240] Negotiation successful
0929 16:45:00.779209 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":333,"threads_started":1}
I20260929 16:45:00.780884 15439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:00.754556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:56723 (local address 127.0.197.126:36941)
0929 16:45:00.755259 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0929 16:45:00.755285 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0929 16:45:00.755943 (+ 658us) server_negotiation.cc:407] Connection header received
0929 16:45:00.758458 (+ 2515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:45:00.758478 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:45:00.759096 (+ 618us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:45:00.759551 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:45:00.761171 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:00.763436 (+ 2265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:00.765440 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:00.766167 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:00.766425 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:00.770320 (+ 3895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:45:00.770335 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:45:00.770360 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:45:00.770420 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:45:00.774990 (+ 4570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:45:00.776071 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:45:00.776085 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:45:00.776096 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:45:00.776373 (+ 277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:45:00.776987 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:45:00.776996 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:45:00.777001 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:45:00.777822 (+ 821us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:45:00.778082 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0929 16:45:00.778986 (+ 904us) server_negotiation.cc:299] Negotiation successful
0929 16:45:00.779544 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":282,"threads_started":1}
I20260929 16:45:00.784765 15437 heartbeater.cc:347] Connected to a master server at 127.0.197.126:36941
I20260929 16:45:00.785163 15437 heartbeater.cc:464] Registering TS with master...
I20260929 16:45:00.786003 15437 heartbeater.cc:511] Master 127.0.197.126:36941 requested a full tablet report, sending...
I20260929 16:45:00.788801 15252 ts_manager.cc:194] Registered new tserver with Master: 4701b5d2213f4db18e70cc60aa1298bd (127.0.197.65:43425)
I20260929 16:45:00.790414 15252 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.65@KRBTEST.COM'} at 127.0.197.65:56723
W20260929 16:45:01.120162 15441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:45:01.120571 15441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:45:01.120759 15441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:45:01.128528 15441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:45:01.128754 15441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:45:01.128960 15441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.66
I20260929 16:45:01.139719 15441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:36245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab
--principal=kudu/127.0.197.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.66:44037
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.197.66
--webserver_port=37485
--tserver_master_addrs=127.0.197.126:36941
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.66
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:45:01.141214 15441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:45:01.142755 15441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:45:01.158641 15446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:45:01.158681 15447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:45:01.158684 15449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:45:01.159507 15441 server_base.cc:1034] running on GCE node
I20260929 16:45:01.160362 15441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:45:01.161561 15441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:45:01.162850 15441 hybrid_clock.cc:648] HybridClock initialized: now 1790700301162779 us; error 67 us; skew 500 ppm
I20260929 16:45:01.163340 15441 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:45:01 dist-test-slave-l31f krb5kdc[14667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:45:01.166846 15441 init.cc:377] Logged in from keytab as kudu/127.0.197.66@KRBTEST.COM (short username kudu)
I20260929 16:45:01.168581 15441 webserver.cc:460] Webserver started at http://127.0.197.66:37485/ using document root <none> and password file <none>
I20260929 16:45:01.169350 15441 fs_manager.cc:359] Metadata directory not provided
I20260929 16:45:01.169476 15441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:45:01.176280 15441 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 16:45:01.179415 15456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:45:01.181151 15441 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 16:45:01.181360 15441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-1/wal
uuid: "023b4695426f4b49a8236ecbc75651b0"
format_stamp: "Formatted at 2026-09-29 16:44:58 on dist-test-slave-l31f"
server_key: "1a95731ce82dbc3a942b69b2f0af378e"
server_key_iv: "edd89e14647e7f8283a0c390d16d397f"
server_key_version: "encryptionkey@0"
I20260929 16:45:01.182665 15441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:45:01.210570 15441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:45:01.216339 15441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:45:01.216683 15441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:45:01.217849 15441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:45:01.219751 15441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:45:01.219869 15441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:45:01.219964 15441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:45:01.220034 15441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:45:01.257321 15441 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.66:44037
I20260929 16:45:01.257383 15568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.66:44037 every 8 connection(s)
I20260929 16:45:01.258988 15441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
I20260929 16:45:01.267369 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 15441
I20260929 16:45:01.268361 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.67:33421
--local_ip_for_outbound_sockets=127.0.197.67
--tserver_master_addrs=127.0.197.126:36941
--webserver_port=42641
--webserver_interface=127.0.197.67
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab
--principal=kudu/127.0.197.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:36245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:45:01 dist-test-slave-l31f krb5kdc[14667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.66@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:45:01.284435 15439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:01.261945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:44015 (local address 127.0.197.126:36941)
0929 16:45:01.262153 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 16:45:01.262158 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:45:01.263867 (+ 1709us) server_negotiation.cc:407] Connection header received
0929 16:45:01.266161 (+ 2294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:45:01.266170 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:45:01.266317 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:45:01.266471 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:45:01.269011 (+ 2540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:01.270370 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:01.272572 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:01.273268 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:01.273466 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:01.277701 (+ 4235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:45:01.277708 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:45:01.277711 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:45:01.277751 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:45:01.280767 (+ 3016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:45:01.281844 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:45:01.281849 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:45:01.281851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:45:01.281996 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:45:01.282479 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:45:01.282489 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:45:01.282491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:45:01.282785 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:45:01.282976 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0929 16:45:01.283796 (+ 820us) server_negotiation.cc:299] Negotiation successful
0929 16:45:01.284067 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 16:45:01.285501 15571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:01.262537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.66:44015)
0929 16:45:01.263673 (+ 1136us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:01.263721 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 16:45:01.265812 (+ 2091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:45:01.266806 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:45:01.266821 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:45:01.267754 (+ 933us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:45:01.268708 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:45:01.268742 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:45:01.270695 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:45:01.270701 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:45:01.272357 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:45:01.272373 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:45:01.273578 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:45:01.273586 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:45:01.273878 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:01.274708 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:45:01.274735 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:45:01.277403 (+ 2668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:45:01.280959 (+ 3556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:45:01.280970 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:45:01.280992 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:45:01.281681 (+ 689us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:45:01.282141 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:45:01.282145 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:45:01.282150 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:45:01.282352 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:45:01.282950 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:45:01.282959 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:45:01.283392 (+ 433us) client_negotiation.cc:769] Sending connection context
0929 16:45:01.283679 (+ 287us) client_negotiation.cc:240] Negotiation successful
0929 16:45:01.284280 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":864,"thread_start_us":522,"threads_started":1}
I20260929 16:45:01.287483 15569 heartbeater.cc:347] Connected to a master server at 127.0.197.126:36941
I20260929 16:45:01.287856 15569 heartbeater.cc:464] Registering TS with master...
I20260929 16:45:01.288836 15569 heartbeater.cc:511] Master 127.0.197.126:36941 requested a full tablet report, sending...
I20260929 16:45:01.290664 15252 ts_manager.cc:194] Registered new tserver with Master: 023b4695426f4b49a8236ecbc75651b0 (127.0.197.66:44037)
I20260929 16:45:01.292076 15252 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.66@KRBTEST.COM'} at 127.0.197.66:44015
W20260929 16:45:01.634040 15572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:45:01.634366 15572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:45:01.634466 15572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:45:01.642351 15572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:45:01.642496 15572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:45:01.642620 15572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.67
I20260929 16:45:01.653002 15572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:36245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab
--principal=kudu/127.0.197.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.67:33421
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.197.67
--webserver_port=42641
--tserver_master_addrs=127.0.197.126:36941
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.67
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:45:01.654387 15572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:45:01.655915 15572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:45:01.671180 15578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:45:01.671232 15580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:45:01.671590 15577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:45:01.671885 15572 server_base.cc:1034] running on GCE node
I20260929 16:45:01.672775 15572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:45:01.673942 15572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:45:01.677381 15572 hybrid_clock.cc:648] HybridClock initialized: now 1790700301677164 us; error 193 us; skew 500 ppm
I20260929 16:45:01.677796 15572 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:45:01 dist-test-slave-l31f krb5kdc[14667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:45:01.681510 15572 init.cc:377] Logged in from keytab as kudu/127.0.197.67@KRBTEST.COM (short username kudu)
I20260929 16:45:01.683436 15572 webserver.cc:460] Webserver started at http://127.0.197.67:42641/ using document root <none> and password file <none>
I20260929 16:45:01.684214 15572 fs_manager.cc:359] Metadata directory not provided
I20260929 16:45:01.684350 15572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:45:01.690853 15572 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 16:45:01.694209 15587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:45:01.696237 15572 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:45:01.696482 15572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-2/wal
uuid: "5553740df9ab445ba3dc8abe4d5b8e58"
format_stamp: "Formatted at 2026-09-29 16:44:58 on dist-test-slave-l31f"
server_key: "5f32c8e03343b339f2d659274da4b283"
server_key_iv: "070c2c94dff4867ddfe64e37252b4dc9"
server_key_version: "encryptionkey@0"
I20260929 16:45:01.697624 15572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:45:01.713304 15572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:45:01.718502 15572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:45:01.718776 15572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:45:01.719944 15572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:45:01.721338 15572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:45:01.721432 15572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:45:01.721516 15572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:45:01.721575 15572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:45:01.761065 15572 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.67:33421
I20260929 16:45:01.761099 15699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.67:33421 every 8 connection(s)
I20260929 16:45:01.762833 15572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
I20260929 16:45:01.768054 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 15572
Sep 29 16:45:01 dist-test-slave-l31f krb5kdc[14667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.67@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:45:01.788394 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:01.769555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53818 (local address 127.0.197.126:36941)
0929 16:45:01.770083 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0929 16:45:01.770088 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:45:01.770103 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:45:01.770155 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:45:01.770159 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:45:01.770219 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:45:01.770361 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:45:01.773242 (+ 2881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:01.774157 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:01.775724 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:01.776174 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:01.776345 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:01.779579 (+ 3234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:45:01.779591 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:45:01.779594 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:45:01.779632 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:45:01.783557 (+ 3925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:45:01.785020 (+ 1463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:45:01.785028 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:45:01.785033 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:45:01.785225 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:45:01.785837 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:45:01.785841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:45:01.785844 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:45:01.786196 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:45:01.787357 (+ 1161us) server_negotiation.cc:1035] Waiting for connection context
0929 16:45:01.787615 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 16:45:01.787892 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":403,"thread_start_us":359,"threads_started":1}
I20260929 16:45:01.790387 15439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:01.766329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:47119 (local address 127.0.197.126:36941)
0929 16:45:01.766619 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0929 16:45:01.766626 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:45:01.768257 (+ 1631us) server_negotiation.cc:407] Connection header received
0929 16:45:01.770896 (+ 2639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:45:01.770901 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:45:01.770986 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:45:01.771116 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:45:01.774020 (+ 2904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:01.774979 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:01.776724 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:01.777182 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:01.777332 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:01.781607 (+ 4275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:45:01.781617 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:45:01.781622 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:45:01.781680 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:45:01.785513 (+ 3833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:45:01.787258 (+ 1745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:45:01.787267 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:45:01.787271 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:45:01.787441 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:45:01.788130 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:45:01.788134 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:45:01.788136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:45:01.788486 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:45:01.788664 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 16:45:01.789741 (+ 1077us) server_negotiation.cc:299] Negotiation successful
0929 16:45:01.789995 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":95}
I20260929 16:45:01.791766 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:01.767043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.67:47119)
0929 16:45:01.768034 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:01.768097 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 16:45:01.770567 (+ 2470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:45:01.772277 (+ 1710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:45:01.772288 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:45:01.773135 (+ 847us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:45:01.773794 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:45:01.773814 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:45:01.775215 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:45:01.775221 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:45:01.776545 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:45:01.776563 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:45:01.777459 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:45:01.777467 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:45:01.777725 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:01.778518 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:45:01.778544 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:45:01.781181 (+ 2637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:45:01.785720 (+ 4539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:45:01.785733 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:45:01.785773 (+ 40us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:45:01.787051 (+ 1278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:45:01.787596 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:45:01.787601 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:45:01.787604 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:45:01.787959 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:45:01.788774 (+ 815us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:45:01.788783 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:45:01.789302 (+ 519us) client_negotiation.cc:769] Sending connection context
0929 16:45:01.789628 (+ 326us) client_negotiation.cc:240] Negotiation successful
0929 16:45:01.790255 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":652,"threads_started":1}
I20260929 16:45:01.793527 15700 heartbeater.cc:347] Connected to a master server at 127.0.197.126:36941
I20260929 16:45:01.793867 15700 heartbeater.cc:464] Registering TS with master...
I20260929 16:45:01.794150 15437 heartbeater.cc:503] Master 127.0.197.126:36941 was elected leader, sending a full tablet report...
I20260929 16:45:01.794878 15700 heartbeater.cc:511] Master 127.0.197.126:36941 requested a full tablet report, sending...
I20260929 16:45:01.796943 15252 ts_manager.cc:194] Registered new tserver with Master: 5553740df9ab445ba3dc8abe4d5b8e58 (127.0.197.67:33421)
I20260929 16:45:01.798200 15252 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.67@KRBTEST.COM'} at 127.0.197.67:47119
I20260929 16:45:01.805572 789 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:45:01.814158 15439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:01.806884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53824 (local address 127.0.197.126:36941)
0929 16:45:01.807278 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0929 16:45:01.807286 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:45:01.807667 (+ 381us) server_negotiation.cc:407] Connection header received
0929 16:45:01.807947 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:45:01.807951 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:45:01.808101 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:45:01.808281 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:45:01.808769 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:01.809604 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:01.811389 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:01.811866 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:01.812046 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:01.812764 (+ 718us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:45:01.813374 (+ 610us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:45:01.813592 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 16:45:01.813729 (+ 137us) server_negotiation.cc:299] Negotiation successful
0929 16:45:01.813835 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260929 16:45:01.815604 15252 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:53824:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:45:01.823524 15252 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:45:01.864828 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:01.840140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47012 (local address 127.0.197.67:33421)
0929 16:45:01.847842 (+ 7702us) server_negotiation.cc:206] Beginning negotiation
0929 16:45:01.847854 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:45:01.847907 (+ 53us) server_negotiation.cc:407] Connection header received
0929 16:45:01.848090 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:45:01.848104 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:45:01.848522 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:45:01.852357 (+ 3835us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:45:01.852808 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:01.856241 (+ 3433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:01.860193 (+ 3952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:01.861832 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:01.863648 (+ 1816us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:01.864048 (+ 400us) server_negotiation.cc:1035] Waiting for connection context
0929 16:45:01.864171 (+ 123us) server_negotiation.cc:299] Negotiation successful
0929 16:45:01.864449 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7444,"thread_start_us":8096,"threads_started":1}
I20260929 16:45:01.865319 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:01.840005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.67:33421 (local address 127.0.0.1:47012)
0929 16:45:01.843684 (+ 3679us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:01.843758 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0929 16:45:01.844078 (+ 320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:45:01.849262 (+ 5184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:45:01.849276 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:45:01.849341 (+ 65us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:45:01.849964 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:45:01.849992 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:45:01.856482 (+ 6490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:45:01.856489 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:45:01.859932 (+ 3443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:45:01.859946 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:45:01.862139 (+ 2193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:45:01.862150 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:45:01.862413 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:01.862500 (+ 87us) client_negotiation.cc:769] Sending connection context
0929 16:45:01.862678 (+ 178us) client_negotiation.cc:240] Negotiation successful
0929 16:45:01.863048 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3261,"mutex_wait_us":56,"thread_start_us":13824,"threads_started":1}
I20260929 16:45:01.865664 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:01.840981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43986 (local address 127.0.197.65:43425)
0929 16:45:01.847134 (+ 6153us) server_negotiation.cc:206] Beginning negotiation
0929 16:45:01.847150 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:45:01.854597 (+ 7447us) server_negotiation.cc:407] Connection header received
0929 16:45:01.855093 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:45:01.855108 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:45:01.855548 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:45:01.855776 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:45:01.857323 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:01.858858 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:01.862065 (+ 3207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:01.863835 (+ 1770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:01.864006 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:01.864262 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0929 16:45:01.864955 (+ 693us) server_negotiation.cc:299] Negotiation successful
0929 16:45:01.865274 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5812,"thread_start_us":3519,"threads_started":1}
I20260929 16:45:01.866400 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:01.840531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:43425 (local address 127.0.0.1:43986)
0929 16:45:01.854357 (+ 13826us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:01.854392 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 16:45:01.854794 (+ 402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:45:01.856149 (+ 1355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:45:01.856154 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:45:01.856212 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:45:01.857000 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:45:01.857026 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:45:01.859089 (+ 2063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:45:01.859095 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:45:01.861604 (+ 2509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:45:01.861630 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:45:01.864469 (+ 2839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:45:01.864476 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:45:01.864705 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:01.864740 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 16:45:01.865933 (+ 1193us) client_negotiation.cc:240] Negotiation successful
0929 16:45:01.866093 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13540,"mutex_wait_us":76,"spinlock_wait_cycles":24960,"thread_start_us":2558,"threads_started":1}
I20260929 16:45:01.868824 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:01.845610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.66:44037 (local address 127.0.0.1:47420)
0929 16:45:01.856058 (+ 10448us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:01.856083 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 16:45:01.856201 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:45:01.859333 (+ 3132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:45:01.859344 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:45:01.859380 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:45:01.860340 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:45:01.860365 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:45:01.862418 (+ 2053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:45:01.862424 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:45:01.865269 (+ 2845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:45:01.865291 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:45:01.867917 (+ 2626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:45:01.867925 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:45:01.868137 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:01.868190 (+ 53us) client_negotiation.cc:769] Sending connection context
0929 16:45:01.868325 (+ 135us) client_negotiation.cc:240] Negotiation successful
0929 16:45:01.868494 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10274,"thread_start_us":10103,"threads_started":1}
I20260929 16:45:01.869513 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:01.845855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47420 (local address 127.0.197.66:44037)
0929 16:45:01.858467 (+ 12612us) server_negotiation.cc:206] Beginning negotiation
0929 16:45:01.858478 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:45:01.858512 (+ 34us) server_negotiation.cc:407] Connection header received
0929 16:45:01.858641 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:45:01.858651 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:45:01.859050 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:45:01.859313 (+ 263us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:45:01.860687 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:01.862223 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:01.865535 (+ 3312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:01.867530 (+ 1995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:01.867823 (+ 293us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:01.868123 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0929 16:45:01.868427 (+ 304us) server_negotiation.cc:299] Negotiation successful
0929 16:45:01.868684 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12390,"thread_start_us":12091,"threads_started":1}
I20260929 16:45:01.870522 15635 tablet_service.cc:1467] Processing CreateTablet for tablet a3020d97d8dd494182bc147544d9424a (DEFAULT_TABLE table=test-table [id=07e7686be75c45e0b4eaf45f46ab5f4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:45:01.872150 15635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3020d97d8dd494182bc147544d9424a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:45:01.871658 15372 tablet_service.cc:1467] Processing CreateTablet for tablet a3020d97d8dd494182bc147544d9424a (DEFAULT_TABLE table=test-table [id=07e7686be75c45e0b4eaf45f46ab5f4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:45:01.873696 15372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3020d97d8dd494182bc147544d9424a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:45:01.874512 15504 tablet_service.cc:1467] Processing CreateTablet for tablet a3020d97d8dd494182bc147544d9424a (DEFAULT_TABLE table=test-table [id=07e7686be75c45e0b4eaf45f46ab5f4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:45:01.876174 15504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3020d97d8dd494182bc147544d9424a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:45:01.890677 15716 tablet_bootstrap.cc:492] T a3020d97d8dd494182bc147544d9424a P 5553740df9ab445ba3dc8abe4d5b8e58: Bootstrap starting.
I20260929 16:45:01.893818 15717 tablet_bootstrap.cc:492] T a3020d97d8dd494182bc147544d9424a P 4701b5d2213f4db18e70cc60aa1298bd: Bootstrap starting.
I20260929 16:45:01.895531 15718 tablet_bootstrap.cc:492] T a3020d97d8dd494182bc147544d9424a P 023b4695426f4b49a8236ecbc75651b0: Bootstrap starting.
I20260929 16:45:01.896525 15716 tablet_bootstrap.cc:654] T a3020d97d8dd494182bc147544d9424a P 5553740df9ab445ba3dc8abe4d5b8e58: Neither blocks nor log segments found. Creating new log.
I20260929 16:45:01.898068 15717 tablet_bootstrap.cc:654] T a3020d97d8dd494182bc147544d9424a P 4701b5d2213f4db18e70cc60aa1298bd: Neither blocks nor log segments found. Creating new log.
I20260929 16:45:01.898113 15716 log.cc:824] T a3020d97d8dd494182bc147544d9424a P 5553740df9ab445ba3dc8abe4d5b8e58: Log is configured to *not* fsync() on all Append() calls
I20260929 16:45:01.898980 15718 tablet_bootstrap.cc:654] T a3020d97d8dd494182bc147544d9424a P 023b4695426f4b49a8236ecbc75651b0: Neither blocks nor log segments found. Creating new log.
I20260929 16:45:01.899497 15717 log.cc:824] T a3020d97d8dd494182bc147544d9424a P 4701b5d2213f4db18e70cc60aa1298bd: Log is configured to *not* fsync() on all Append() calls
I20260929 16:45:01.900640 15718 log.cc:824] T a3020d97d8dd494182bc147544d9424a P 023b4695426f4b49a8236ecbc75651b0: Log is configured to *not* fsync() on all Append() calls
I20260929 16:45:01.904201 15718 tablet_bootstrap.cc:492] T a3020d97d8dd494182bc147544d9424a P 023b4695426f4b49a8236ecbc75651b0: No bootstrap required, opened a new log
I20260929 16:45:01.904232 15717 tablet_bootstrap.cc:492] T a3020d97d8dd494182bc147544d9424a P 4701b5d2213f4db18e70cc60aa1298bd: No bootstrap required, opened a new log
I20260929 16:45:01.904232 15716 tablet_bootstrap.cc:492] T a3020d97d8dd494182bc147544d9424a P 5553740df9ab445ba3dc8abe4d5b8e58: No bootstrap required, opened a new log
I20260929 16:45:01.904596 15718 ts_tablet_manager.cc:1397] T a3020d97d8dd494182bc147544d9424a P 023b4695426f4b49a8236ecbc75651b0: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 16:45:01.904603 15716 ts_tablet_manager.cc:1397] T a3020d97d8dd494182bc147544d9424a P 5553740df9ab445ba3dc8abe4d5b8e58: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20260929 16:45:01.904601 15717 ts_tablet_manager.cc:1397] T a3020d97d8dd494182bc147544d9424a P 4701b5d2213f4db18e70cc60aa1298bd: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260929 16:45:01.912618 15718 raft_consensus.cc:348] T a3020d97d8dd494182bc147544d9424a P 023b4695426f4b49a8236ecbc75651b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5553740df9ab445ba3dc8abe4d5b8e58" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33421 } } peers { permanent_uuid: "4701b5d2213f4db18e70cc60aa1298bd" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43425 } } peers { permanent_uuid: "023b4695426f4b49a8236ecbc75651b0" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44037 } }
I20260929 16:45:01.913206 15718 raft_consensus.cc:374] T a3020d97d8dd494182bc147544d9424a P 023b4695426f4b49a8236ecbc75651b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:45:01.913311 15718 raft_consensus.cc:729] T a3020d97d8dd494182bc147544d9424a P 023b4695426f4b49a8236ecbc75651b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 023b4695426f4b49a8236ecbc75651b0, State: Initialized, Role: FOLLOWER
I20260929 16:45:01.914572 15718 consensus_queue.cc:260] T a3020d97d8dd494182bc147544d9424a P 023b4695426f4b49a8236ecbc75651b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5553740df9ab445ba3dc8abe4d5b8e58" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33421 } } peers { permanent_uuid: "4701b5d2213f4db18e70cc60aa1298bd" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43425 } } peers { permanent_uuid: "023b4695426f4b49a8236ecbc75651b0" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44037 } }
I20260929 16:45:01.915848 15569 heartbeater.cc:503] Master 127.0.197.126:36941 was elected leader, sending a full tablet report...
I20260929 16:45:01.916260 15716 raft_consensus.cc:348] T a3020d97d8dd494182bc147544d9424a P 5553740df9ab445ba3dc8abe4d5b8e58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5553740df9ab445ba3dc8abe4d5b8e58" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33421 } } peers { permanent_uuid: "4701b5d2213f4db18e70cc60aa1298bd" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43425 } } peers { permanent_uuid: "023b4695426f4b49a8236ecbc75651b0" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44037 } }
I20260929 16:45:01.916260 15717 raft_consensus.cc:348] T a3020d97d8dd494182bc147544d9424a P 4701b5d2213f4db18e70cc60aa1298bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5553740df9ab445ba3dc8abe4d5b8e58" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33421 } } peers { permanent_uuid: "4701b5d2213f4db18e70cc60aa1298bd" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43425 } } peers { permanent_uuid: "023b4695426f4b49a8236ecbc75651b0" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44037 } }
I20260929 16:45:01.916639 15718 ts_tablet_manager.cc:1428] T a3020d97d8dd494182bc147544d9424a P 023b4695426f4b49a8236ecbc75651b0: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
I20260929 16:45:01.916730 15716 raft_consensus.cc:374] T a3020d97d8dd494182bc147544d9424a P 5553740df9ab445ba3dc8abe4d5b8e58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:45:01.916730 15717 raft_consensus.cc:374] T a3020d97d8dd494182bc147544d9424a P 4701b5d2213f4db18e70cc60aa1298bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:45:01.916829 15716 raft_consensus.cc:729] T a3020d97d8dd494182bc147544d9424a P 5553740df9ab445ba3dc8abe4d5b8e58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5553740df9ab445ba3dc8abe4d5b8e58, State: Initialized, Role: FOLLOWER
I20260929 16:45:01.916841 15717 raft_consensus.cc:729] T a3020d97d8dd494182bc147544d9424a P 4701b5d2213f4db18e70cc60aa1298bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4701b5d2213f4db18e70cc60aa1298bd, State: Initialized, Role: FOLLOWER
I20260929 16:45:01.917506 15716 consensus_queue.cc:260] T a3020d97d8dd494182bc147544d9424a P 5553740df9ab445ba3dc8abe4d5b8e58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5553740df9ab445ba3dc8abe4d5b8e58" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33421 } } peers { permanent_uuid: "4701b5d2213f4db18e70cc60aa1298bd" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43425 } } peers { permanent_uuid: "023b4695426f4b49a8236ecbc75651b0" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44037 } }
I20260929 16:45:01.917506 15717 consensus_queue.cc:260] T a3020d97d8dd494182bc147544d9424a P 4701b5d2213f4db18e70cc60aa1298bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5553740df9ab445ba3dc8abe4d5b8e58" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33421 } } peers { permanent_uuid: "4701b5d2213f4db18e70cc60aa1298bd" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43425 } } peers { permanent_uuid: "023b4695426f4b49a8236ecbc75651b0" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44037 } }
I20260929 16:45:01.919601 15716 ts_tablet_manager.cc:1428] T a3020d97d8dd494182bc147544d9424a P 5553740df9ab445ba3dc8abe4d5b8e58: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260929 16:45:01.919791 15717 ts_tablet_manager.cc:1428] T a3020d97d8dd494182bc147544d9424a P 4701b5d2213f4db18e70cc60aa1298bd: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260929 16:45:01.920212 15700 heartbeater.cc:503] Master 127.0.197.126:36941 was elected leader, sending a full tablet report...
W20260929 16:45:02.004169 15438 tablet.cc:2369] T a3020d97d8dd494182bc147544d9424a P 4701b5d2213f4db18e70cc60aa1298bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:45:02.011452 15570 tablet.cc:2369] T a3020d97d8dd494182bc147544d9424a P 023b4695426f4b49a8236ecbc75651b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:45:02.017554 15701 tablet.cc:2369] T a3020d97d8dd494182bc147544d9424a P 5553740df9ab445ba3dc8abe4d5b8e58: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:45:02.017962 15724 raft_consensus.cc:482] T a3020d97d8dd494182bc147544d9424a P 5553740df9ab445ba3dc8abe4d5b8e58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:45:02.018302 15724 raft_consensus.cc:504] T a3020d97d8dd494182bc147544d9424a P 5553740df9ab445ba3dc8abe4d5b8e58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5553740df9ab445ba3dc8abe4d5b8e58" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33421 } } peers { permanent_uuid: "4701b5d2213f4db18e70cc60aa1298bd" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43425 } } peers { permanent_uuid: "023b4695426f4b49a8236ecbc75651b0" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44037 } }
I20260929 16:45:02.020246 15724 leader_election.cc:288] T a3020d97d8dd494182bc147544d9424a P 5553740df9ab445ba3dc8abe4d5b8e58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4701b5d2213f4db18e70cc60aa1298bd (127.0.197.65:43425), 023b4695426f4b49a8236ecbc75651b0 (127.0.197.66:44037)
I20260929 16:45:02.026111 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:02.020472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:43425 (local address 127.0.197.67:51431)
0929 16:45:02.020693 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:02.020711 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:45:02.020869 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:45:02.021453 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:45:02.021458 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:45:02.021486 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:45:02.021681 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:45:02.021690 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:45:02.022724 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:45:02.022730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:45:02.024134 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:45:02.024147 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:45:02.025322 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:45:02.025328 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:45:02.025525 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:02.025549 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 16:45:02.025670 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 16:45:02.025817 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260929 16:45:02.026091 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:02.020814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:51431 (local address 127.0.197.65:43425)
0929 16:45:02.021058 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0929 16:45:02.021068 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:45:02.021095 (+ 27us) server_negotiation.cc:407] Connection header received
0929 16:45:02.021215 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:45:02.021225 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:45:02.021309 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:45:02.021413 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:45:02.021845 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:02.022559 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:02.024320 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:02.025135 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:02.025332 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:02.025414 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 16:45:02.025699 (+ 285us) server_negotiation.cc:299] Negotiation successful
0929 16:45:02.025837 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 16:45:02.027287 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:02.020887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.66:44037 (local address 127.0.197.67:54867)
0929 16:45:02.021507 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:02.021528 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:45:02.021697 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:45:02.022044 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:45:02.022049 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:45:02.022074 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:45:02.022416 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:45:02.022424 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:45:02.023673 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:45:02.023679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:45:02.025104 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:45:02.025116 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:45:02.026477 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:45:02.026487 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:45:02.026690 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:02.026722 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 16:45:02.026824 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 16:45:02.026953 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":330,"threads_started":1}
I20260929 16:45:02.027400 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:02.021264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:54867 (local address 127.0.197.66:44037)
0929 16:45:02.021478 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 16:45:02.021486 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:45:02.021630 (+ 144us) server_negotiation.cc:407] Connection header received
0929 16:45:02.021840 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:45:02.021846 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:45:02.021919 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:45:02.022031 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:45:02.022579 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:02.023478 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:02.025293 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:02.026093 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:02.026300 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:02.026394 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 16:45:02.026925 (+ 531us) server_negotiation.cc:299] Negotiation successful
0929 16:45:02.027133 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 16:45:02.027575 15392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3020d97d8dd494182bc147544d9424a" candidate_uuid: "5553740df9ab445ba3dc8abe4d5b8e58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4701b5d2213f4db18e70cc60aa1298bd" is_pre_election: true
I20260929 16:45:02.028218 15392 raft_consensus.cc:2454] T a3020d97d8dd494182bc147544d9424a P 4701b5d2213f4db18e70cc60aa1298bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5553740df9ab445ba3dc8abe4d5b8e58 in term 0.
I20260929 16:45:02.028569 15524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3020d97d8dd494182bc147544d9424a" candidate_uuid: "5553740df9ab445ba3dc8abe4d5b8e58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "023b4695426f4b49a8236ecbc75651b0" is_pre_election: true
I20260929 16:45:02.029152 15524 raft_consensus.cc:2454] T a3020d97d8dd494182bc147544d9424a P 023b4695426f4b49a8236ecbc75651b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5553740df9ab445ba3dc8abe4d5b8e58 in term 0.
I20260929 16:45:02.029421 15591 leader_election.cc:302] T a3020d97d8dd494182bc147544d9424a P 5553740df9ab445ba3dc8abe4d5b8e58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4701b5d2213f4db18e70cc60aa1298bd, 5553740df9ab445ba3dc8abe4d5b8e58; no voters:
I20260929 16:45:02.030100 15724 raft_consensus.cc:2781] T a3020d97d8dd494182bc147544d9424a P 5553740df9ab445ba3dc8abe4d5b8e58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:45:02.030299 15724 raft_consensus.cc:482] T a3020d97d8dd494182bc147544d9424a P 5553740df9ab445ba3dc8abe4d5b8e58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:45:02.030422 15724 raft_consensus.cc:3037] T a3020d97d8dd494182bc147544d9424a P 5553740df9ab445ba3dc8abe4d5b8e58 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:45:02.032855 15724 raft_consensus.cc:504] T a3020d97d8dd494182bc147544d9424a P 5553740df9ab445ba3dc8abe4d5b8e58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5553740df9ab445ba3dc8abe4d5b8e58" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33421 } } peers { permanent_uuid: "4701b5d2213f4db18e70cc60aa1298bd" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43425 } } peers { permanent_uuid: "023b4695426f4b49a8236ecbc75651b0" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44037 } }
I20260929 16:45:02.033596 15724 leader_election.cc:288] T a3020d97d8dd494182bc147544d9424a P 5553740df9ab445ba3dc8abe4d5b8e58 [CANDIDATE]: Term 1 election: Requested vote from peers 4701b5d2213f4db18e70cc60aa1298bd (127.0.197.65:43425), 023b4695426f4b49a8236ecbc75651b0 (127.0.197.66:44037)
I20260929 16:45:02.034591 15524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3020d97d8dd494182bc147544d9424a" candidate_uuid: "5553740df9ab445ba3dc8abe4d5b8e58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "023b4695426f4b49a8236ecbc75651b0"
I20260929 16:45:02.034682 15392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3020d97d8dd494182bc147544d9424a" candidate_uuid: "5553740df9ab445ba3dc8abe4d5b8e58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4701b5d2213f4db18e70cc60aa1298bd"
I20260929 16:45:02.034823 15524 raft_consensus.cc:3037] T a3020d97d8dd494182bc147544d9424a P 023b4695426f4b49a8236ecbc75651b0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:45:02.035084 15392 raft_consensus.cc:3037] T a3020d97d8dd494182bc147544d9424a P 4701b5d2213f4db18e70cc60aa1298bd [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:45:02.037504 15524 raft_consensus.cc:2454] T a3020d97d8dd494182bc147544d9424a P 023b4695426f4b49a8236ecbc75651b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5553740df9ab445ba3dc8abe4d5b8e58 in term 1.
I20260929 16:45:02.038125 15591 leader_election.cc:302] T a3020d97d8dd494182bc147544d9424a P 5553740df9ab445ba3dc8abe4d5b8e58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 023b4695426f4b49a8236ecbc75651b0, 5553740df9ab445ba3dc8abe4d5b8e58; no voters:
I20260929 16:45:02.038324 15392 raft_consensus.cc:2454] T a3020d97d8dd494182bc147544d9424a P 4701b5d2213f4db18e70cc60aa1298bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5553740df9ab445ba3dc8abe4d5b8e58 in term 1.
I20260929 16:45:02.038549 15724 raft_consensus.cc:2781] T a3020d97d8dd494182bc147544d9424a P 5553740df9ab445ba3dc8abe4d5b8e58 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:45:02.039287 15724 raft_consensus.cc:686] T a3020d97d8dd494182bc147544d9424a P 5553740df9ab445ba3dc8abe4d5b8e58 [term 1 LEADER]: Becoming Leader. State: Replica: 5553740df9ab445ba3dc8abe4d5b8e58, State: Running, Role: LEADER
I20260929 16:45:02.039878 15724 consensus_queue.cc:237] T a3020d97d8dd494182bc147544d9424a P 5553740df9ab445ba3dc8abe4d5b8e58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5553740df9ab445ba3dc8abe4d5b8e58" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33421 } } peers { permanent_uuid: "4701b5d2213f4db18e70cc60aa1298bd" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43425 } } peers { permanent_uuid: "023b4695426f4b49a8236ecbc75651b0" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44037 } }
I20260929 16:45:02.045104 15251 catalog_manager.cc:5495] T a3020d97d8dd494182bc147544d9424a P 5553740df9ab445ba3dc8abe4d5b8e58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5553740df9ab445ba3dc8abe4d5b8e58 (127.0.197.67). New cstate: current_term: 1 leader_uuid: "5553740df9ab445ba3dc8abe4d5b8e58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5553740df9ab445ba3dc8abe4d5b8e58" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 33421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4701b5d2213f4db18e70cc60aa1298bd" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "023b4695426f4b49a8236ecbc75651b0" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44037 } health_report { overall_health: UNKNOWN } } }
I20260929 16:45:02.104278 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:02.097845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47016 (local address 127.0.197.67:33421)
0929 16:45:02.098079 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 16:45:02.098084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:45:02.098191 (+ 107us) server_negotiation.cc:407] Connection header received
0929 16:45:02.098368 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:45:02.098372 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:45:02.098464 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:45:02.098618 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:45:02.098992 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:02.099846 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:02.101394 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:02.101822 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:02.101927 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:02.102820 (+ 893us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:45:02.103657 (+ 837us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:45:02.103838 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 16:45:02.103975 (+ 137us) server_negotiation.cc:299] Negotiation successful
0929 16:45:02.104059 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260929 16:45:02.113956 15392 raft_consensus.cc:1261] T a3020d97d8dd494182bc147544d9424a P 4701b5d2213f4db18e70cc60aa1298bd [term 1 FOLLOWER]: Refusing update from remote peer 5553740df9ab445ba3dc8abe4d5b8e58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:45:02.113960 15524 raft_consensus.cc:1261] T a3020d97d8dd494182bc147544d9424a P 023b4695426f4b49a8236ecbc75651b0 [term 1 FOLLOWER]: Refusing update from remote peer 5553740df9ab445ba3dc8abe4d5b8e58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:45:02.115463 15724 consensus_queue.cc:1035] T a3020d97d8dd494182bc147544d9424a P 5553740df9ab445ba3dc8abe4d5b8e58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4701b5d2213f4db18e70cc60aa1298bd" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 43425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:45:02.115835 15726 consensus_queue.cc:1035] T a3020d97d8dd494182bc147544d9424a P 5553740df9ab445ba3dc8abe4d5b8e58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "023b4695426f4b49a8236ecbc75651b0" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 44037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:45:02.129858 15731 mvcc.cc:205] Tried to move back new op lower bound from 7334708437444399104 to 7334708437159243776. Current Snapshot: MvccSnapshot[applied={T|T < 7334708437444399104}]
I20260929 16:45:02.144891 15251 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:53824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:45:02.145190 15251 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:53824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:45:02.149595 15251 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644: Sending DeleteTablet for 3 replicas of tablet a3020d97d8dd494182bc147544d9424a
I20260929 16:45:02.151208 15635 tablet_service.cc:1514] Processing DeleteTablet for tablet a3020d97d8dd494182bc147544d9424a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:45:02 UTC) from {username='kudu', principal='kudu/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:47012
I20260929 16:45:02.151234 15504 tablet_service.cc:1514] Processing DeleteTablet for tablet a3020d97d8dd494182bc147544d9424a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:45:02 UTC) from {username='kudu', principal='kudu/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:47420
I20260929 16:45:02.151470 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 15289
I20260929 16:45:02.152750 15372 tablet_service.cc:1514] Processing DeleteTablet for tablet a3020d97d8dd494182bc147544d9424a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:45:02 UTC) from {username='kudu', principal='kudu/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:43986
I20260929 16:45:02.164921 15739 tablet_replica.cc:330] T a3020d97d8dd494182bc147544d9424a P 023b4695426f4b49a8236ecbc75651b0: stopping tablet replica
I20260929 16:45:02.165630 15739 raft_consensus.cc:2229] T a3020d97d8dd494182bc147544d9424a P 023b4695426f4b49a8236ecbc75651b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:45:02.166607 15739 raft_consensus.cc:2258] T a3020d97d8dd494182bc147544d9424a P 023b4695426f4b49a8236ecbc75651b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:45:02.167678 15738 tablet_replica.cc:330] T a3020d97d8dd494182bc147544d9424a P 5553740df9ab445ba3dc8abe4d5b8e58: stopping tablet replica
I20260929 16:45:02.168144 15740 tablet_replica.cc:330] T a3020d97d8dd494182bc147544d9424a P 4701b5d2213f4db18e70cc60aa1298bd: stopping tablet replica
I20260929 16:45:02.168241 15738 raft_consensus.cc:2229] T a3020d97d8dd494182bc147544d9424a P 5553740df9ab445ba3dc8abe4d5b8e58 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:45:02.168744 15740 raft_consensus.cc:2229] T a3020d97d8dd494182bc147544d9424a P 4701b5d2213f4db18e70cc60aa1298bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:45:02.169051 15738 raft_consensus.cc:2258] T a3020d97d8dd494182bc147544d9424a P 5553740df9ab445ba3dc8abe4d5b8e58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:45:02.169406 15740 raft_consensus.cc:2258] T a3020d97d8dd494182bc147544d9424a P 4701b5d2213f4db18e70cc60aa1298bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:45:02.171567 15740 ts_tablet_manager.cc:1905] T a3020d97d8dd494182bc147544d9424a P 4701b5d2213f4db18e70cc60aa1298bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:45:02.171567 15739 ts_tablet_manager.cc:1905] T a3020d97d8dd494182bc147544d9424a P 023b4695426f4b49a8236ecbc75651b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:45:02.176329 15738 ts_tablet_manager.cc:1905] T a3020d97d8dd494182bc147544d9424a P 5553740df9ab445ba3dc8abe4d5b8e58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:45:02.180081 15740 ts_tablet_manager.cc:1918] T a3020d97d8dd494182bc147544d9424a P 4701b5d2213f4db18e70cc60aa1298bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:45:02.180078 15739 ts_tablet_manager.cc:1918] T a3020d97d8dd494182bc147544d9424a P 023b4695426f4b49a8236ecbc75651b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:45:02.180284 15739 log.cc:1196] T a3020d97d8dd494182bc147544d9424a P 023b4695426f4b49a8236ecbc75651b0: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-1/wal/wals/a3020d97d8dd494182bc147544d9424a
I20260929 16:45:02.180477 15740 log.cc:1196] T a3020d97d8dd494182bc147544d9424a P 4701b5d2213f4db18e70cc60aa1298bd: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-0/wal/wals/a3020d97d8dd494182bc147544d9424a
I20260929 16:45:02.180944 15739 ts_tablet_manager.cc:1939] T a3020d97d8dd494182bc147544d9424a P 023b4695426f4b49a8236ecbc75651b0: Deleting consensus metadata
I20260929 16:45:02.181229 15740 ts_tablet_manager.cc:1939] T a3020d97d8dd494182bc147544d9424a P 4701b5d2213f4db18e70cc60aa1298bd: Deleting consensus metadata
I20260929 16:45:02.183272 15239 catalog_manager.cc:4841] TS 023b4695426f4b49a8236ecbc75651b0 (127.0.197.66:44037): tablet a3020d97d8dd494182bc147544d9424a (table test-table [id=07e7686be75c45e0b4eaf45f46ab5f4f]) successfully deleted
I20260929 16:45:02.183411 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:02.154047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43990 (local address 127.0.197.65:43425)
0929 16:45:02.154246 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 16:45:02.154252 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:45:02.154274 (+ 22us) server_negotiation.cc:407] Connection header received
0929 16:45:02.154343 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:45:02.154349 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:45:02.154445 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:45:02.154556 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:45:02.156791 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:02.157981 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:02.160332 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:02.161056 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:02.161201 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:02.167970 (+ 6769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:45:02.167992 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:45:02.168014 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:45:02.168073 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:45:02.173068 (+ 4995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:45:02.174228 (+ 1160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:45:02.174238 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:45:02.174244 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:45:02.174474 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:45:02.180037 (+ 5563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:45:02.180045 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:45:02.180049 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:45:02.180693 (+ 644us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:45:02.181051 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0929 16:45:02.182612 (+ 1561us) server_negotiation.cc:299] Negotiation successful
0929 16:45:02.182937 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 16:45:02.183660 15239 catalog_manager.cc:4841] TS 4701b5d2213f4db18e70cc60aa1298bd (127.0.197.65:43425): tablet a3020d97d8dd494182bc147544d9424a (table test-table [id=07e7686be75c45e0b4eaf45f46ab5f4f]) successfully deleted
I20260929 16:45:02.184121 15432 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:45:02.189671 15738 ts_tablet_manager.cc:1918] T a3020d97d8dd494182bc147544d9424a P 5553740df9ab445ba3dc8abe4d5b8e58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:45:02.189919 15738 log.cc:1196] T a3020d97d8dd494182bc147544d9424a P 5553740df9ab445ba3dc8abe4d5b8e58: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-2/wal/wals/a3020d97d8dd494182bc147544d9424a
I20260929 16:45:02.190485 15738 ts_tablet_manager.cc:1939] T a3020d97d8dd494182bc147544d9424a P 5553740df9ab445ba3dc8abe4d5b8e58: Deleting consensus metadata
I20260929 16:45:02.192314 15238 catalog_manager.cc:4841] TS 5553740df9ab445ba3dc8abe4d5b8e58 (127.0.197.67:33421): tablet a3020d97d8dd494182bc147544d9424a (table test-table [id=07e7686be75c45e0b4eaf45f46ab5f4f]) successfully deleted
I20260929 16:45:02.336321 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 15289
I20260929 16:45:02.356137 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 15441
I20260929 16:45:02.371799 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:02.357030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47434 (local address 127.0.197.66:44037)
0929 16:45:02.357205 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 16:45:02.357213 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:45:02.357259 (+ 46us) server_negotiation.cc:407] Connection header received
0929 16:45:02.357485 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:45:02.357490 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:45:02.357583 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:45:02.357706 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:45:02.359093 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:02.359851 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:02.361200 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:02.361611 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:02.361778 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:02.365090 (+ 3312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:45:02.365106 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:45:02.365122 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:45:02.365166 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:45:02.368554 (+ 3388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:45:02.369394 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:45:02.369414 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:45:02.369416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:45:02.369551 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:45:02.369978 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:45:02.369982 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:45:02.369985 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:45:02.370476 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:45:02.370785 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0929 16:45:02.371167 (+ 382us) server_negotiation.cc:299] Negotiation successful
0929 16:45:02.371411 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 16:45:02.372713 15564 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:45:02.487620 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 15441
I20260929 16:45:02.512099 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 15572
I20260929 16:45:02.530861 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:02.512995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47024 (local address 127.0.197.67:33421)
0929 16:45:02.513180 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 16:45:02.513185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:45:02.513203 (+ 18us) server_negotiation.cc:407] Connection header received
0929 16:45:02.513447 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:45:02.513452 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:45:02.513600 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:45:02.513693 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:45:02.515656 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:02.516512 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:02.518455 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:02.519155 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:02.519271 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:02.523210 (+ 3939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:45:02.523224 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:45:02.523240 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:45:02.523292 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:45:02.527339 (+ 4047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:45:02.528424 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:45:02.528436 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:45:02.528443 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:45:02.528617 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:45:02.529074 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:45:02.529077 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:45:02.529079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:45:02.529436 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:45:02.529678 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0929 16:45:02.530164 (+ 486us) server_negotiation.cc:299] Negotiation successful
0929 16:45:02.530367 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 16:45:02.532033 15695 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:45:02.661306 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 15572
I20260929 16:45:02.682595 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 15219
I20260929 16:45:02.684011 15282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:45:02.804828 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 15219
I20260929 16:45:02.828831 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:36941
--webserver_interface=127.0.197.126
--webserver_port=44179
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:36245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.126:36941
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:45:03.191440 15745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:45:03.191910 15745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:45:03.192082 15745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:45:03.200234 15745 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:45:03.200426 15745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:45:03.200546 15745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:45:03.200644 15745 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:45:03.200685 15745 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:45:03.211493 15745 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:36245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.197.126:36941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:36941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.197.126
--webserver_port=44179
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:45:03.212935 15745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:45:03.214273 15745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:45:03.227461 15751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:45:03.227452 15753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:45:03.227522 15750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:45:03.228833 15745 server_base.cc:1034] running on GCE node
I20260929 16:45:03.229725 15745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:45:03.231354 15745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:45:03.232599 15745 hybrid_clock.cc:648] HybridClock initialized: now 1790700303232525 us; error 66 us; skew 500 ppm
I20260929 16:45:03.233013 15745 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:45:03 dist-test-slave-l31f krb5kdc[14667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700303, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:45:03.242652 15745 init.cc:377] Logged in from keytab as kudu/127.0.197.126@KRBTEST.COM (short username kudu)
I20260929 16:45:03.244729 15745 webserver.cc:460] Webserver started at http://127.0.197.126:44179/ using document root <none> and password file <none>
I20260929 16:45:03.245599 15745 fs_manager.cc:359] Metadata directory not provided
I20260929 16:45:03.245790 15745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:45:03.252380 15745 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 16:45:03.257222 15760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:45:03.259382 15745 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260929 16:45:03.259649 15745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "54cae7cc4b3b4fd5adff223d09b60644"
format_stamp: "Formatted at 2026-09-29 16:44:56 on dist-test-slave-l31f"
server_key: "dc88f0acbb71b42de1e581a465a44380"
server_key_iv: "03d41cedf079e423cfc4bc7198ee5f6d"
server_key_version: "encryptionkey@0"
I20260929 16:45:03.260804 15745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:45:03.286477 15745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:45:03.291769 15745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:45:03.292127 15745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:45:03.314808 15745 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.126:36941
I20260929 16:45:03.314808 15811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.126:36941 every 8 connection(s)
I20260929 16:45:03.316936 15745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:45:03.326020 15812 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:45:03.327207 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 15745
I20260929 16:45:03.328085 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.65:43425
--local_ip_for_outbound_sockets=127.0.197.65
--tserver_master_addrs=127.0.197.126:36941
--webserver_port=46237
--webserver_interface=127.0.197.65
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab
--principal=kudu/127.0.197.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:36245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:45:03.330760 15812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644: Bootstrap starting.
I20260929 16:45:03.355620 15812 log.cc:824] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644: Log is configured to *not* fsync() on all Append() calls
I20260929 16:45:03.377616 15812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:45:03.378268 15812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644: Bootstrap complete.
I20260929 16:45:03.389298 15812 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54cae7cc4b3b4fd5adff223d09b60644" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 36941 } }
I20260929 16:45:03.390355 15812 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54cae7cc4b3b4fd5adff223d09b60644, State: Initialized, Role: FOLLOWER
I20260929 16:45:03.391207 15812 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54cae7cc4b3b4fd5adff223d09b60644" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 36941 } }
I20260929 16:45:03.391520 15812 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:45:03.391664 15812 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:45:03.391868 15812 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644 [term 2 FOLLOWER]: Advancing to term 3
I20260929 16:45:03.396768 15812 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54cae7cc4b3b4fd5adff223d09b60644" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 36941 } }
I20260929 16:45:03.397377 15812 leader_election.cc:302] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54cae7cc4b3b4fd5adff223d09b60644; no voters:
I20260929 16:45:03.398142 15812 leader_election.cc:288] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 16:45:03.398309 15816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 16:45:03.399613 15816 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644 [term 3 LEADER]: Becoming Leader. State: Replica: 54cae7cc4b3b4fd5adff223d09b60644, State: Running, Role: LEADER
I20260929 16:45:03.400252 15816 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54cae7cc4b3b4fd5adff223d09b60644" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 36941 } }
I20260929 16:45:03.400700 15812 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:45:03.403133 15818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54cae7cc4b3b4fd5adff223d09b60644. Latest consensus state: current_term: 3 leader_uuid: "54cae7cc4b3b4fd5adff223d09b60644" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54cae7cc4b3b4fd5adff223d09b60644" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 36941 } } }
I20260929 16:45:03.403237 15817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "54cae7cc4b3b4fd5adff223d09b60644" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54cae7cc4b3b4fd5adff223d09b60644" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 36941 } } }
I20260929 16:45:03.403837 15818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644 [sys.catalog]: This master's current role is: LEADER
I20260929 16:45:03.403888 15817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644 [sys.catalog]: This master's current role is: LEADER
I20260929 16:45:03.418368 15832 catalog_manager.cc:1259] Loaded cluster ID: 6abb2c0bcc98481f9103fab29127bfbd
I20260929 16:45:03.418519 15832 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644: loading cluster ID for follower catalog manager: success
W20260929 16:45:03.420209 15832 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 54cae7cc4b3b4fd5adff223d09b60644: acquiring CA information for follower catalog manager: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
W20260929 16:45:03.420383 15832 catalog_manager.cc:873] Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO: failed to prepare follower catalog manager, will retry
I20260929 16:45:03.420692 15821 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:45:03.425272 15821 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:45:03.426440 15821 catalog_manager.cc:1259] Loaded cluster ID: 6abb2c0bcc98481f9103fab29127bfbd
I20260929 16:45:03.426580 15821 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260929 16:45:03.427910 15821 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb08f1589ad google::LogMessage::Fail() at ??:0
@ 0x7fb08f15b88a google::LogMessage::SendToLog() at ??:0
@ 0x7fb08f1584de google::LogMessage::Flush() at ??:0
@ 0x7fb08f159a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb0a4b8b512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fb0a4b88976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fb0a4c10610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fb0a4c10439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb091da4345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb091da791c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fb091da7799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb091d82eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb09505c6db start_thread at ??:0
@ 0x7fb08dff261f clone at ??:0
W20260929 16:45:03.737718 15814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:45:03.738121 15814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:45:03.738297 15814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:45:03.746891 15814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:45:03.747144 15814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:45:03.747335 15814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.65
I20260929 16:45:03.757979 15814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:36245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab
--principal=kudu/127.0.197.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.65:43425
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.197.65
--webserver_port=46237
--tserver_master_addrs=127.0.197.126:36941
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.65
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:45:03.759744 15814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:45:03.761262 15814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:45:03.777068 15856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:45:03.777143 15854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:45:03.777944 15814 server_base.cc:1034] running on GCE node
W20260929 16:45:03.778012 15853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:45:03.778752 15814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:45:03.779940 15814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:45:03.781188 15814 hybrid_clock.cc:648] HybridClock initialized: now 1790700303781115 us; error 59 us; skew 500 ppm
I20260929 16:45:03.781585 15814 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:45:03 dist-test-slave-l31f krb5kdc[14667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700303, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:45:03.785233 15814 init.cc:377] Logged in from keytab as kudu/127.0.197.65@KRBTEST.COM (short username kudu)
I20260929 16:45:03.786887 15814 webserver.cc:460] Webserver started at http://127.0.197.65:46237/ using document root <none> and password file <none>
I20260929 16:45:03.787752 15814 fs_manager.cc:359] Metadata directory not provided
I20260929 16:45:03.787871 15814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:45:03.793846 15814 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:45:03.797443 15863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:45:03.799446 15814 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260929 16:45:03.799677 15814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-0/wal
uuid: "4701b5d2213f4db18e70cc60aa1298bd"
format_stamp: "Formatted at 2026-09-29 16:44:57 on dist-test-slave-l31f"
server_key: "8e4e3938bdcc38fa9f103ea5f3bc03ab"
server_key_iv: "65c157d9d03ce08f0fa7590964447678"
server_key_version: "encryptionkey@0"
I20260929 16:45:03.800791 15814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:45:03.836838 15814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:45:03.841787 15814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:45:03.842154 15814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:45:03.843349 15814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:45:03.844901 15814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:45:03.845010 15814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:45:03.845176 15814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:45:03.845265 15814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:45:03.885699 15814 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.65:43425
I20260929 16:45:03.885749 15975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.65:43425 every 8 connection(s)
I20260929 16:45:03.887562 15814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
I20260929 16:45:03.893427 15978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:03.890955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.65:38231)
0929 16:45:03.892088 (+ 1133us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:03.892410 (+ 322us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":356,"threads_started":1}
W20260929 16:45:03.894379 15976 heartbeater.cc:644] Failed to heartbeat to 127.0.197.126:36941 (0 consecutive failures): Network error: Failed to ping master at 127.0.197.126:36941: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
I20260929 16:45:03.895576 15978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:03.895134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.65:38145)
0929 16:45:03.895341 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:03.895449 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 16:45:03.896557 15978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:03.896165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.65:54275)
0929 16:45:03.896334 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:03.896425 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260929 16:45:03.896863 15976 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 16:45:03.896881 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 15814
I20260929 16:45:03.897594 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.66:44037
--local_ip_for_outbound_sockets=127.0.197.66
--tserver_master_addrs=127.0.197.126:36941
--webserver_port=37485
--webserver_interface=127.0.197.66
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab
--principal=kudu/127.0.197.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:36245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:45:04.253074 15979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:45:04.253402 15979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:45:04.253589 15979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:45:04.261469 15979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:45:04.261615 15979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:45:04.261825 15979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.66
I20260929 16:45:04.272586 15979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:36245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab
--principal=kudu/127.0.197.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.66:44037
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.197.66
--webserver_port=37485
--tserver_master_addrs=127.0.197.126:36941
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.66
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:45:04.274065 15979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:45:04.275625 15979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:45:04.292333 15985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:45:04.292354 15987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:45:04.293097 15984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:45:04.293862 15979 server_base.cc:1034] running on GCE node
I20260929 16:45:04.294739 15979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:45:04.296005 15979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:45:04.297237 15979 hybrid_clock.cc:648] HybridClock initialized: now 1790700304297200 us; error 37 us; skew 500 ppm
I20260929 16:45:04.297643 15979 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:45:04 dist-test-slave-l31f krb5kdc[14667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700304, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:45:04.301203 15979 init.cc:377] Logged in from keytab as kudu/127.0.197.66@KRBTEST.COM (short username kudu)
I20260929 16:45:04.303076 15979 webserver.cc:460] Webserver started at http://127.0.197.66:37485/ using document root <none> and password file <none>
I20260929 16:45:04.303885 15979 fs_manager.cc:359] Metadata directory not provided
I20260929 16:45:04.304000 15979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:45:04.310165 15979 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260929 16:45:04.313350 15994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:45:04.315245 15979 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:45:04.315426 15979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-1/wal
uuid: "023b4695426f4b49a8236ecbc75651b0"
format_stamp: "Formatted at 2026-09-29 16:44:58 on dist-test-slave-l31f"
server_key: "1a95731ce82dbc3a942b69b2f0af378e"
server_key_iv: "edd89e14647e7f8283a0c390d16d397f"
server_key_version: "encryptionkey@0"
I20260929 16:45:04.316582 15979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:45:04.354902 15979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:45:04.360556 15979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:45:04.360846 15979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:45:04.361891 15979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:45:04.363499 15979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:45:04.363605 15979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:45:04.363687 15979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:45:04.363744 15979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:45:04.400225 15979 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.66:44037
I20260929 16:45:04.400287 16106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.66:44037 every 8 connection(s)
I20260929 16:45:04.401947 15979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
I20260929 16:45:04.405874 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 15979
I20260929 16:45:04.406694 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.67:33421
--local_ip_for_outbound_sockets=127.0.197.67
--tserver_master_addrs=127.0.197.126:36941
--webserver_port=42641
--webserver_interface=127.0.197.67
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab
--principal=kudu/127.0.197.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:36245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:45:04.407815 16109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:04.405508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.66:51001)
0929 16:45:04.406459 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:04.406741 (+ 282us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":412,"threads_started":1}
W20260929 16:45:04.408658 16107 heartbeater.cc:644] Failed to heartbeat to 127.0.197.126:36941 (0 consecutive failures): Network error: Failed to ping master at 127.0.197.126:36941: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
I20260929 16:45:04.409667 16109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:04.409272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.66:59099)
0929 16:45:04.409456 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:04.409544 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 16:45:04.410908 16109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:04.410385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.66:37145)
0929 16:45:04.410608 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:04.410726 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260929 16:45:04.411290 16107 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 16:45:04.757539 16110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:45:04.757858 16110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:45:04.758049 16110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:45:04.765811 16110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:45:04.765961 16110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:45:04.766194 16110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.67
I20260929 16:45:04.776952 16110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:36245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab
--principal=kudu/127.0.197.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.67:33421
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.197.67
--webserver_port=42641
--tserver_master_addrs=127.0.197.126:36941
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.67
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:45:04.778420 16110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:45:04.779970 16110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:45:04.795194 16116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:45:04.795245 16118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:45:04.795116 16115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:45:04.796605 16110 server_base.cc:1034] running on GCE node
I20260929 16:45:04.797353 16110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:45:04.798476 16110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:45:04.799813 16110 hybrid_clock.cc:648] HybridClock initialized: now 1790700304799758 us; error 80 us; skew 500 ppm
I20260929 16:45:04.800196 16110 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:45:04 dist-test-slave-l31f krb5kdc[14667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700304, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:45:04.804963 16110 init.cc:377] Logged in from keytab as kudu/127.0.197.67@KRBTEST.COM (short username kudu)
I20260929 16:45:04.806888 16110 webserver.cc:460] Webserver started at http://127.0.197.67:42641/ using document root <none> and password file <none>
I20260929 16:45:04.807720 16110 fs_manager.cc:359] Metadata directory not provided
I20260929 16:45:04.807891 16110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:45:04.814095 16110 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 16:45:04.817324 16125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:45:04.819298 16110 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:45:04.819521 16110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-2/wal
uuid: "5553740df9ab445ba3dc8abe4d5b8e58"
format_stamp: "Formatted at 2026-09-29 16:44:58 on dist-test-slave-l31f"
server_key: "5f32c8e03343b339f2d659274da4b283"
server_key_iv: "070c2c94dff4867ddfe64e37252b4dc9"
server_key_version: "encryptionkey@0"
I20260929 16:45:04.820642 16110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:45:04.839908 16110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:45:04.845382 16110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:45:04.845824 16110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:45:04.847100 16110 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:45:04.848836 16110 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:45:04.848979 16110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:45:04.849135 16110 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:45:04.849236 16110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:45:04.889740 16110 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.67:33421
I20260929 16:45:04.889786 16237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.67:33421 every 8 connection(s)
I20260929 16:45:04.891489 16110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
I20260929 16:45:04.894094 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 16110
I20260929 16:45:04.897469 16241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:04.895061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.67:50109)
0929 16:45:04.896072 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:04.896369 (+ 297us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":688,"thread_start_us":477,"threads_started":1}
W20260929 16:45:04.898253 16238 heartbeater.cc:644] Failed to heartbeat to 127.0.197.126:36941 (0 consecutive failures): Network error: Failed to ping master at 127.0.197.126:36941: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
I20260929 16:45:04.898420 16242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:04.897435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.65:55783)
0929 16:45:04.898004 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:04.898114 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":304,"threads_started":1}
I20260929 16:45:04.899269 16241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:04.898847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.67:44427)
0929 16:45:04.898991 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:04.899162 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 16:45:04.900249 16241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:04.899892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.67:37133)
0929 16:45:04.900066 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:04.900141 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260929 16:45:04.900645 16238 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 16:45:05.413264 16243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:05.412107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.66:44849)
0929 16:45:05.412804 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:05.412934 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":288,"threads_started":1}
I20260929 16:45:05.900334 16244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:05.899498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.65:35711)
0929 16:45:05.900063 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:05.900152 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":244,"threads_started":1}
I20260929 16:45:05.902124 16245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:05.901210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.67:54889)
0929 16:45:05.901736 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:05.901843 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":279,"threads_started":1}
I20260929 16:45:06.415732 16246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:06.414800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.66:34219)
0929 16:45:06.415443 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:06.415554 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":379,"threads_started":1}
I20260929 16:45:06.902374 16247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:06.901413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.65:40055)
0929 16:45:06.902100 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:06.902193 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":384,"threads_started":1}
I20260929 16:45:06.904304 16248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:06.903387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.67:38641)
0929 16:45:06.903935 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:06.904091 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":241,"threads_started":1}
I20260929 16:45:07.417817 16249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:07.416884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.66:33201)
0929 16:45:07.417372 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:07.417462 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":271,"threads_started":1}
I20260929 16:45:07.904567 16250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:07.903659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.65:60555)
0929 16:45:07.904255 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:07.904357 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":287,"threads_started":1}
I20260929 16:45:07.906150 16251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:07.905329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.67:47601)
0929 16:45:07.905878 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:07.905975 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":323,"threads_started":1}
I20260929 16:45:08.419831 16252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:08.418918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.66:43893)
0929 16:45:08.419544 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:08.419642 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":350,"threads_started":1}
I20260929 16:45:08.906651 16253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:08.905877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.65:54891)
0929 16:45:08.906373 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:08.906463 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":260,"threads_started":1}
I20260929 16:45:08.908030 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:08.907247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.67:53501)
0929 16:45:08.907784 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:08.907864 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":306,"threads_started":1}
I20260929 16:45:09.421883 16255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:09.420970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.66:42747)
0929 16:45:09.421598 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:09.421694 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":280,"threads_started":1}
I20260929 16:45:09.908448 16256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:09.907662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.65:60833)
0929 16:45:09.908177 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:09.908278 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":254,"threads_started":1}
I20260929 16:45:09.910152 16257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:09.909111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.67:41997)
0929 16:45:09.909869 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:09.909951 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":544,"threads_started":1}
I20260929 16:45:10.423816 16258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:10.422976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.66:39457)
0929 16:45:10.423530 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:10.423649 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":257,"threads_started":1}
I20260929 16:45:10.910554 16259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:10.909623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.65:51573)
0929 16:45:10.910213 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:10.910314 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":291,"threads_started":1}
I20260929 16:45:10.912240 16260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:10.911333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.67:60231)
0929 16:45:10.911907 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:10.912042 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":337,"threads_started":1}
I20260929 16:45:11.425941 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:11.424923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.66:40019)
0929 16:45:11.425631 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:11.425730 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":413,"threads_started":1}
I20260929 16:45:11.912401 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:11.911667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.65:58773)
0929 16:45:11.912133 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:11.912222 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":241,"threads_started":1}
I20260929 16:45:11.913929 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:11.913166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.67:59157)
0929 16:45:11.913637 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:11.913731 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":210,"threads_started":1}
I20260929 16:45:12.427917 16264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:12.427191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.66:52869)
0929 16:45:12.427659 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:12.427750 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":236,"threads_started":1}
I20260929 16:45:12.914323 16265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:12.913440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.65:50133)
0929 16:45:12.913982 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:12.914115 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":263,"threads_started":1}
I20260929 16:45:12.915743 16266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:12.914981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.67:51339)
0929 16:45:12.915493 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:12.915581 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":230,"threads_started":1}
I20260929 16:45:13.429770 16267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:13.428938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.66:46717)
0929 16:45:13.429455 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:13.429576 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":246,"threads_started":1}
I20260929 16:45:13.916388 16268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:13.915523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.65:53259)
0929 16:45:13.916028 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:13.916166 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":314,"threads_started":1}
I20260929 16:45:13.917536 16269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:13.916746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.67:46363)
0929 16:45:13.917287 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:13.917365 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":277,"threads_started":1}
I20260929 16:45:14.431665 16270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:14.430872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.66:47807)
0929 16:45:14.431391 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:14.431483 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":314,"threads_started":1}
I20260929 16:45:14.918402 16271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:14.917531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.65:44831)
0929 16:45:14.918067 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:14.918190 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":237,"threads_started":1}
I20260929 16:45:14.919286 16272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:14.918519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.67:44593)
0929 16:45:14.919057 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:14.919146 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":262,"threads_started":1}
I20260929 16:45:15.433594 16273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:15.432763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.66:32849)
0929 16:45:15.433301 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:15.433405 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":271,"threads_started":1}
I20260929 16:45:15.920378 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:15.919564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.65:35853)
0929 16:45:15.920113 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:15.920213 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":305,"threads_started":1}
I20260929 16:45:15.920964 16275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:15.920160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.67:50821)
0929 16:45:15.920655 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:15.920787 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":262,"threads_started":1}
I20260929 16:45:16.435708 16276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:16.434831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.66:34573)
0929 16:45:16.435454 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:16.435543 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":394,"threads_started":1}
I20260929 16:45:16.922487 16277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:16.921478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.65:42537)
0929 16:45:16.922129 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:16.922283 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":329,"threads_started":1}
I20260929 16:45:16.922514 16278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:16.921756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.67:35273)
0929 16:45:16.922286 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:16.922374 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":307,"threads_started":1}
I20260929 16:45:17.437525 16279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:17.436675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.66:53423)
0929 16:45:17.437214 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:17.437323 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":257,"threads_started":1}
I20260929 16:45:17.924674 16280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:17.923689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.65:42791)
0929 16:45:17.924308 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:17.924422 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":353,"threads_started":1}
I20260929 16:45:17.924695 16281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:17.923701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.67:43769)
0929 16:45:17.924308 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:17.924432 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":356,"threads_started":1}
I20260929 16:45:18.439518 16282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:18.438692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.66:54831)
0929 16:45:18.439229 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:18.439324 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":232,"threads_started":1}
I20260929 16:45:18.926785 16283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:18.926010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.67:36971)
0929 16:45:18.926515 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:18.926604 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":273,"threads_started":1}
I20260929 16:45:18.927295 16284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:18.926133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.65:34509)
0929 16:45:18.926956 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:18.927119 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":599,"threads_started":1}
I20260929 16:45:19.441607 16285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:19.440856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.66:56043)
0929 16:45:19.441321 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:19.441416 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":246,"threads_started":1}
I20260929 16:45:19.906709 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 15814
I20260929 16:45:19.927425 16286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:19.907891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51672 (local address 127.0.197.65:43425)
0929 16:45:19.908582 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0929 16:45:19.908600 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 16:45:19.908642 (+ 42us) server_negotiation.cc:407] Connection header received
0929 16:45:19.911106 (+ 2464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:45:19.911118 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:45:19.911597 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:45:19.911983 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:45:19.913620 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:19.914867 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:19.916231 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:19.916951 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:19.917182 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:19.920167 (+ 2985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:45:19.920175 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:45:19.920194 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:45:19.920254 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:45:19.923531 (+ 3277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:45:19.924363 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:45:19.924371 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:45:19.924379 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:45:19.924632 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:45:19.925061 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:45:19.925069 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:45:19.925073 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:45:19.925730 (+ 657us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:45:19.926143 (+ 413us) server_negotiation.cc:1035] Waiting for connection context
0929 16:45:19.926514 (+ 371us) server_negotiation.cc:299] Negotiation successful
0929 16:45:19.926995 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":333,"threads_started":1}
I20260929 16:45:19.928872 16287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:19.927966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.67:56527)
0929 16:45:19.928512 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:19.928638 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":266,"threads_started":1}
I20260929 16:45:19.929311 15971 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:45:20.014546 16288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:19.928075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:36941 (local address 127.0.197.65:39785)
0929 16:45:19.928812 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:19.929202 (+ 390us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":463,"threads_started":1}
I20260929 16:45:20.015972 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 15814
I20260929 16:45:20.034984 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 15979
I20260929 16:45:20.053720 16290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:20.036081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57440 (local address 127.0.197.66:44037)
0929 16:45:20.036660 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0929 16:45:20.036675 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:45:20.036712 (+ 37us) server_negotiation.cc:407] Connection header received
0929 16:45:20.038289 (+ 1577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:45:20.038299 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:45:20.038739 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:45:20.039139 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:45:20.040575 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:20.041812 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:20.043251 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:20.044010 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:20.044160 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:20.046956 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:45:20.046966 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:45:20.046984 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:45:20.047142 (+ 158us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:45:20.050142 (+ 3000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:45:20.050982 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:45:20.050990 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:45:20.050998 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:45:20.051270 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:45:20.051624 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:45:20.051627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:45:20.051629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:45:20.052266 (+ 637us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:45:20.052535 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0929 16:45:20.052935 (+ 400us) server_negotiation.cc:299] Negotiation successful
0929 16:45:20.053348 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":304,"threads_started":1}
I20260929 16:45:20.055240 16102 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:45:20.140887 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 15979
I20260929 16:45:20.159291 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 16110
I20260929 16:45:20.179891 16292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:20.160446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49278 (local address 127.0.197.67:33421)
0929 16:45:20.161126 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0929 16:45:20.161141 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:45:20.161183 (+ 42us) server_negotiation.cc:407] Connection header received
0929 16:45:20.163362 (+ 2179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:45:20.163378 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:45:20.163909 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:45:20.164250 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:45:20.165759 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:20.167147 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:20.168566 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:20.169327 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:20.169492 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:20.172627 (+ 3135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:45:20.172637 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:45:20.172672 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:45:20.172724 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:45:20.176168 (+ 3444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:45:20.177100 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:45:20.177108 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:45:20.177115 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:45:20.177316 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:45:20.177716 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:45:20.177719 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:45:20.177721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:45:20.178325 (+ 604us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:45:20.178562 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 16:45:20.179071 (+ 509us) server_negotiation.cc:299] Negotiation successful
0929 16:45:20.179572 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":320,"threads_started":1}
I20260929 16:45:20.181443 16233 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:45:20.267472 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 16110
2026-09-29T16:45:20Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (25206 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:45:20 dist-test-slave-l31f krb5kdc[16296](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:45:20 dist-test-slave-l31f krb5kdc[16296](info): set up 2 sockets
Sep 29 16:45:20 dist-test-slave-l31f krb5kdc[16296](info): commencing operation
krb5kdc: starting...
W20260929 16:45:21.396097 789 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/test-user.keytab.
Sep 29 16:45:21 dist-test-slave-l31f krb5kdc[16296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:45:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:45:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.197.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
I20260929 16:45:21.671588 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:37141
--webserver_interface=127.0.197.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:33025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.126:37141
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:45:22.034914 16309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:45:22.035367 16309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:45:22.035562 16309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:45:22.043360 16309 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 16:45:22.043478 16309 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 16:45:22.043543 16309 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 16:45:22.043624 16309 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:45:22.043723 16309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:45:22.043852 16309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:45:22.043941 16309 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:45:22.044024 16309 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:45:22.054538 16309 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:33025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.197.126:37141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:37141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.197.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:45:22.056036 16309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:45:22.057466 16309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:45:22.070797 16314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:45:22.070773 16315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:45:22.070856 16317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:45:22.071605 16309 server_base.cc:1034] running on GCE node
I20260929 16:45:22.072435 16309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:45:22.074024 16309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:45:22.075237 16309 hybrid_clock.cc:648] HybridClock initialized: now 1790700322075168 us; error 43 us; skew 500 ppm
I20260929 16:45:22.075584 16309 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:45:22 dist-test-slave-l31f krb5kdc[16296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:45:22.084774 16309 init.cc:377] Logged in from keytab as kudu/127.0.197.126@KRBTEST.COM (short username kudu)
I20260929 16:45:22.086567 16309 webserver.cc:460] Webserver started at http://127.0.197.126:35897/ using document root <none> and password file <none>
I20260929 16:45:22.087446 16309 fs_manager.cc:359] Metadata directory not provided
I20260929 16:45:22.087574 16309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:45:22.087921 16309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:45:22.091280 16309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/master-0/data/instance:
uuid: "5014d3abe04b487cab15866468715f4e"
format_stamp: "Formatted at 2026-09-29 16:45:22 on dist-test-slave-l31f"
server_key: "37b457cf171f0a8a8f28d7261010da4e"
server_key_iv: "996c493fa1f8594a13099f8a85ac83a5"
server_key_version: "encryptionkey@0"
I20260929 16:45:22.092185 16309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/master-0/wal/instance:
uuid: "5014d3abe04b487cab15866468715f4e"
format_stamp: "Formatted at 2026-09-29 16:45:22 on dist-test-slave-l31f"
server_key: "37b457cf171f0a8a8f28d7261010da4e"
server_key_iv: "996c493fa1f8594a13099f8a85ac83a5"
server_key_version: "encryptionkey@0"
I20260929 16:45:22.097975 16309 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:45:22.102072 16324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:45:22.104200 16309 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 16:45:22.104583 16309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "5014d3abe04b487cab15866468715f4e"
format_stamp: "Formatted at 2026-09-29 16:45:22 on dist-test-slave-l31f"
server_key: "37b457cf171f0a8a8f28d7261010da4e"
server_key_iv: "996c493fa1f8594a13099f8a85ac83a5"
server_key_version: "encryptionkey@0"
I20260929 16:45:22.104916 16309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:45:22.130144 16309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:45:22.135701 16309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:45:22.136024 16309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:45:22.159332 16309 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.126:37141
I20260929 16:45:22.159367 16375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.126:37141 every 8 connection(s)
I20260929 16:45:22.160943 16309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:45:22.165370 16376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:45:22.168268 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 16309
I20260929 16:45:22.168512 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/master-0/wal/instance
I20260929 16:45:22.169333 789 external_mini_cluster.cc:1428] Setting key 1d9e7de53d3520a0a502fd0c3a3af064
I20260929 16:45:22.178543 16376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e: Bootstrap starting.
I20260929 16:45:22.181942 16376 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e: Neither blocks nor log segments found. Creating new log.
I20260929 16:45:22.183130 16376 log.cc:824] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:45:22 dist-test-slave-l31f krb5kdc[16296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:45:22.186646 16376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e: No bootstrap required, opened a new log
I20260929 16:45:22.196049 16379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:22.171871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40786 (local address 127.0.197.126:37141)
0929 16:45:22.172829 (+ 958us) server_negotiation.cc:206] Beginning negotiation
0929 16:45:22.172843 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:45:22.172888 (+ 45us) server_negotiation.cc:407] Connection header received
0929 16:45:22.174671 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:45:22.174698 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:45:22.175216 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:45:22.175650 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:45:22.177300 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:22.178699 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:22.180224 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:22.180957 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:22.181187 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:22.185891 (+ 4704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:45:22.185905 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:45:22.185930 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:45:22.185986 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:45:22.190520 (+ 4534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:45:22.191603 (+ 1083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:45:22.191618 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:45:22.191628 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:45:22.191877 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:45:22.192316 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:45:22.192323 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:45:22.192328 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:45:22.193175 (+ 847us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:45:22.193496 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0929 16:45:22.193909 (+ 413us) server_negotiation.cc:299] Negotiation successful
0929 16:45:22.194440 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":703,"thread_start_us":384,"threads_started":1}
I20260929 16:45:22.197877 16376 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5014d3abe04b487cab15866468715f4e" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 37141 } }
I20260929 16:45:22.198310 16376 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:45:22.198417 16376 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5014d3abe04b487cab15866468715f4e, State: Initialized, Role: FOLLOWER
I20260929 16:45:22.199149 16376 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5014d3abe04b487cab15866468715f4e" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 37141 } }
I20260929 16:45:22.199432 16376 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:45:22.199607 16376 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:45:22.199800 16376 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:45:22.202167 16376 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5014d3abe04b487cab15866468715f4e" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 37141 } }
I20260929 16:45:22.202759 16376 leader_election.cc:302] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5014d3abe04b487cab15866468715f4e; no voters:
I20260929 16:45:22.203509 16376 leader_election.cc:288] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:45:22.203783 16381 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:45:22.204865 16381 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e [term 1 LEADER]: Becoming Leader. State: Replica: 5014d3abe04b487cab15866468715f4e, State: Running, Role: LEADER
I20260929 16:45:22.205574 16381 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5014d3abe04b487cab15866468715f4e" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 37141 } }
I20260929 16:45:22.205993 16376 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:45:22.208664 16382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5014d3abe04b487cab15866468715f4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5014d3abe04b487cab15866468715f4e" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 37141 } } }
I20260929 16:45:22.209229 16383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5014d3abe04b487cab15866468715f4e. Latest consensus state: current_term: 1 leader_uuid: "5014d3abe04b487cab15866468715f4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5014d3abe04b487cab15866468715f4e" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 37141 } } }
I20260929 16:45:22.209610 16383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e [sys.catalog]: This master's current role is: LEADER
I20260929 16:45:22.209582 16382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e [sys.catalog]: This master's current role is: LEADER
I20260929 16:45:22.219456 16390 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:45:22.225210 16390 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:45:22.235994 16390 catalog_manager.cc:1347] Generated new cluster ID: d7396f1a6df84581bbb913b096f068a6
I20260929 16:45:22.236153 16390 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:45:22.253926 16390 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:45:22.256052 16390 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:45:22.278607 16390 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e: Generated new TSK 0
I20260929 16:45:22.279853 16390 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.197.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab.
Entry for principal kudu/127.0.197.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab.
I20260929 16:45:22.360778 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.65:0
--local_ip_for_outbound_sockets=127.0.197.65
--webserver_interface=127.0.197.65
--webserver_port=0
--tserver_master_addrs=127.0.197.126:37141
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab
--principal=kudu/127.0.197.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:33025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:45:22.749188 16402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:45:22.749512 16402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:45:22.749693 16402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:45:22.757309 16402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:45:22.757450 16402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:45:22.757664 16402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.65
I20260929 16:45:22.768105 16402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:33025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab
--principal=kudu/127.0.197.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.197.65
--webserver_port=0
--tserver_master_addrs=127.0.197.126:37141
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.65
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:45:22.769536 16402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:45:22.771086 16402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:45:22.786108 16407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:45:22.786052 16408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:45:22.786063 16410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:45:22.786933 16402 server_base.cc:1034] running on GCE node
I20260929 16:45:22.787685 16402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:45:22.788841 16402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:45:22.790076 16402 hybrid_clock.cc:648] HybridClock initialized: now 1790700322790000 us; error 64 us; skew 500 ppm
I20260929 16:45:22.790494 16402 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:45:22 dist-test-slave-l31f krb5kdc[16296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:45:22.793823 16402 init.cc:377] Logged in from keytab as kudu/127.0.197.65@KRBTEST.COM (short username kudu)
I20260929 16:45:22.795480 16402 webserver.cc:460] Webserver started at http://127.0.197.65:43127/ using document root <none> and password file <none>
I20260929 16:45:22.796180 16402 fs_manager.cc:359] Metadata directory not provided
I20260929 16:45:22.796285 16402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:45:22.796554 16402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:45:22.799727 16402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-0/data/instance:
uuid: "a1e2ec51ef5d43cbb18df0696c0d159d"
format_stamp: "Formatted at 2026-09-29 16:45:22 on dist-test-slave-l31f"
server_key: "6cc5d59ebdd86d718375e75d3ed52042"
server_key_iv: "33506d4e02c0d8aced7de2559addef2a"
server_key_version: "encryptionkey@0"
I20260929 16:45:22.800611 16402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-0/wal/instance:
uuid: "a1e2ec51ef5d43cbb18df0696c0d159d"
format_stamp: "Formatted at 2026-09-29 16:45:22 on dist-test-slave-l31f"
server_key: "6cc5d59ebdd86d718375e75d3ed52042"
server_key_iv: "33506d4e02c0d8aced7de2559addef2a"
server_key_version: "encryptionkey@0"
I20260929 16:45:22.806524 16402 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 16:45:22.810760 16417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:45:22.812526 16402 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:45:22.812759 16402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-0/wal
uuid: "a1e2ec51ef5d43cbb18df0696c0d159d"
format_stamp: "Formatted at 2026-09-29 16:45:22 on dist-test-slave-l31f"
server_key: "6cc5d59ebdd86d718375e75d3ed52042"
server_key_iv: "33506d4e02c0d8aced7de2559addef2a"
server_key_version: "encryptionkey@0"
I20260929 16:45:22.812992 16402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:45:22.846237 16402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:45:22.851094 16402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:45:22.851501 16402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:45:22.852993 16402 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:45:22.855366 16402 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:45:22.855557 16402 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:45:22.855746 16402 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:45:22.855835 16402 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:45:22.892197 16402 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.65:41857
I20260929 16:45:22.892234 16529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.65:41857 every 8 connection(s)
I20260929 16:45:22.893584 16402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
I20260929 16:45:22.902803 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 16402
I20260929 16:45:22.903118 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-0/wal/instance
I20260929 16:45:22.903759 789 external_mini_cluster.cc:1428] Setting key 46efffb497f2475ba95fcd7714ff0a68
Sep 29 16:45:22 dist-test-slave-l31f krb5kdc[16296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.65@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:45:22.917960 16532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:22.896365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:38113 (local address 127.0.197.126:37141)
0929 16:45:22.896842 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0929 16:45:22.896847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:45:22.897891 (+ 1044us) server_negotiation.cc:407] Connection header received
0929 16:45:22.899885 (+ 1994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:45:22.899889 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:45:22.899962 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:45:22.900097 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:45:22.901715 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:22.902593 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:22.904645 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:22.905174 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:22.905288 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:22.909870 (+ 4582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:45:22.909880 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:45:22.909888 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:45:22.909942 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:45:22.914007 (+ 4065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:45:22.915029 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:45:22.915038 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:45:22.915043 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:45:22.915228 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:45:22.915743 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:45:22.915752 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:45:22.915758 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:45:22.916166 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:45:22.916327 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 16:45:22.917178 (+ 851us) server_negotiation.cc:299] Negotiation successful
0929 16:45:22.917469 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":243,"threads_started":1}
I20260929 16:45:22.918929 16533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:22.896983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.65:38113)
0929 16:45:22.897709 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:22.897754 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 16:45:22.899597 (+ 1843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:45:22.900312 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:45:22.900321 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:45:22.900866 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:45:22.901536 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:45:22.901554 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:45:22.902783 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:45:22.902787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:45:22.904429 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:45:22.904447 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:45:22.905539 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:45:22.905547 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:45:22.905946 (+ 399us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:22.906913 (+ 967us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:45:22.906947 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:45:22.909532 (+ 2585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:45:22.914276 (+ 4744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:45:22.914286 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:45:22.914303 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:45:22.914815 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:45:22.915374 (+ 559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:45:22.915378 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:45:22.915381 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:45:22.915599 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:45:22.916302 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:45:22.916308 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:45:22.916736 (+ 428us) client_negotiation.cc:769] Sending connection context
0929 16:45:22.916987 (+ 251us) client_negotiation.cc:240] Negotiation successful
0929 16:45:22.917551 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":292,"threads_started":1}
I20260929 16:45:22.920764 16530 heartbeater.cc:347] Connected to a master server at 127.0.197.126:37141
I20260929 16:45:22.921116 16530 heartbeater.cc:464] Registering TS with master...
I20260929 16:45:22.921890 16530 heartbeater.cc:511] Master 127.0.197.126:37141 requested a full tablet report, sending...
I20260929 16:45:22.924226 16341 ts_manager.cc:194] Registered new tserver with Master: a1e2ec51ef5d43cbb18df0696c0d159d (127.0.197.65:41857)
I20260929 16:45:22.925732 16341 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.65@KRBTEST.COM'} at 127.0.197.65:38113
WARNING: no policy specified for kudu/127.0.197.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab.
Entry for principal kudu/127.0.197.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab.
I20260929 16:45:22.964031 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.66:0
--local_ip_for_outbound_sockets=127.0.197.66
--webserver_interface=127.0.197.66
--webserver_port=0
--tserver_master_addrs=127.0.197.126:37141
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab
--principal=kudu/127.0.197.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:33025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:45:23.325022 16536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:45:23.325366 16536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:45:23.325466 16536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:45:23.333823 16536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:45:23.333974 16536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:45:23.334110 16536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.66
I20260929 16:45:23.344815 16536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:33025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab
--principal=kudu/127.0.197.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.197.66
--webserver_port=0
--tserver_master_addrs=127.0.197.126:37141
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.66
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:45:23.346131 16536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:45:23.347564 16536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:45:23.363035 16541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:45:23.363035 16544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:45:23.362947 16542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:45:23.364665 16536 server_base.cc:1034] running on GCE node
I20260929 16:45:23.365345 16536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:45:23.366386 16536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:45:23.367605 16536 hybrid_clock.cc:648] HybridClock initialized: now 1790700323367558 us; error 46 us; skew 500 ppm
I20260929 16:45:23.367969 16536 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:45:23 dist-test-slave-l31f krb5kdc[16296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700323, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:45:23.371381 16536 init.cc:377] Logged in from keytab as kudu/127.0.197.66@KRBTEST.COM (short username kudu)
I20260929 16:45:23.373258 16536 webserver.cc:460] Webserver started at http://127.0.197.66:40259/ using document root <none> and password file <none>
I20260929 16:45:23.374044 16536 fs_manager.cc:359] Metadata directory not provided
I20260929 16:45:23.374212 16536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:45:23.374552 16536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:45:23.377817 16536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-1/data/instance:
uuid: "ddc41210f2dc4a3189be59eea46a2e31"
format_stamp: "Formatted at 2026-09-29 16:45:23 on dist-test-slave-l31f"
server_key: "0ecface9c95a52c3d3d9898c4d68dc2c"
server_key_iv: "8173df979e7f0113ead6a4a34c678444"
server_key_version: "encryptionkey@0"
I20260929 16:45:23.378696 16536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-1/wal/instance:
uuid: "ddc41210f2dc4a3189be59eea46a2e31"
format_stamp: "Formatted at 2026-09-29 16:45:23 on dist-test-slave-l31f"
server_key: "0ecface9c95a52c3d3d9898c4d68dc2c"
server_key_iv: "8173df979e7f0113ead6a4a34c678444"
server_key_version: "encryptionkey@0"
I20260929 16:45:23.384593 16536 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 16:45:23.388574 16551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:45:23.390170 16536 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:45:23.390475 16536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-1/wal
uuid: "ddc41210f2dc4a3189be59eea46a2e31"
format_stamp: "Formatted at 2026-09-29 16:45:23 on dist-test-slave-l31f"
server_key: "0ecface9c95a52c3d3d9898c4d68dc2c"
server_key_iv: "8173df979e7f0113ead6a4a34c678444"
server_key_version: "encryptionkey@0"
I20260929 16:45:23.390717 16536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:45:23.413950 16536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:45:23.418973 16536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:45:23.419381 16536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:45:23.420456 16536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:45:23.421913 16536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:45:23.422034 16536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:45:23.422209 16536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:45:23.422286 16536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:45:23.457253 16536 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.66:43983
I20260929 16:45:23.457304 16663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.66:43983 every 8 connection(s)
I20260929 16:45:23.458712 16536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
I20260929 16:45:23.460947 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 16536
I20260929 16:45:23.461362 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-1/wal/instance
I20260929 16:45:23.462025 789 external_mini_cluster.cc:1428] Setting key 24e586c3e37078e9f9f3a3a66742f606
Sep 29 16:45:23 dist-test-slave-l31f krb5kdc[16296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700323, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.66@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:45:23.483187 16666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:23.462286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:50683 (local address 127.0.197.126:37141)
0929 16:45:23.462879 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0929 16:45:23.462887 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:45:23.464167 (+ 1280us) server_negotiation.cc:407] Connection header received
0929 16:45:23.466351 (+ 2184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:45:23.466356 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:45:23.466456 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:45:23.466594 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:45:23.468384 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:23.469157 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:23.470952 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:23.471493 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:23.471648 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:23.476159 (+ 4511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:45:23.476165 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:45:23.476169 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:45:23.476210 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:45:23.479130 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:45:23.480332 (+ 1202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:45:23.480336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:45:23.480339 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:45:23.480475 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:45:23.481071 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:45:23.481075 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:45:23.481077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:45:23.481373 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:45:23.481507 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 16:45:23.482606 (+ 1099us) server_negotiation.cc:299] Negotiation successful
0929 16:45:23.482806 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":278,"threads_started":1}
I20260929 16:45:23.484932 16667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:23.462982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.66:50683)
0929 16:45:23.463942 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:23.464001 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 16:45:23.465853 (+ 1852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:45:23.466870 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:45:23.466880 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:45:23.467559 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:45:23.468177 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:45:23.468201 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:45:23.469351 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:45:23.469357 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:45:23.470743 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:45:23.470760 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:45:23.471759 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:45:23.471767 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:45:23.472130 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:23.472922 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:45:23.472951 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:45:23.475762 (+ 2811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:45:23.479330 (+ 3568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:45:23.479344 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:45:23.479368 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:45:23.480153 (+ 785us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:45:23.480606 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:45:23.480614 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:45:23.480620 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:45:23.480914 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:45:23.481516 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:45:23.481528 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:45:23.482072 (+ 544us) client_negotiation.cc:769] Sending connection context
0929 16:45:23.482442 (+ 370us) client_negotiation.cc:240] Negotiation successful
0929 16:45:23.483190 (+ 748us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":334,"threads_started":1}
I20260929 16:45:23.486786 16664 heartbeater.cc:347] Connected to a master server at 127.0.197.126:37141
WARNING: no policy specified for kudu/127.0.197.67@KRBTEST.COM; defaulting to no policy
I20260929 16:45:23.487249 16664 heartbeater.cc:464] Registering TS with master...
I20260929 16:45:23.488072 16664 heartbeater.cc:511] Master 127.0.197.126:37141 requested a full tablet report, sending...
I20260929 16:45:23.490109 16341 ts_manager.cc:194] Registered new tserver with Master: ddc41210f2dc4a3189be59eea46a2e31 (127.0.197.66:43983)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.67@KRBTEST.COM" created.
I20260929 16:45:23.491356 16341 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.66@KRBTEST.COM'} at 127.0.197.66:50683
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab.
Entry for principal kudu/127.0.197.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab.
I20260929 16:45:23.523489 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.67:0
--local_ip_for_outbound_sockets=127.0.197.67
--webserver_interface=127.0.197.67
--webserver_port=0
--tserver_master_addrs=127.0.197.126:37141
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab
--principal=kudu/127.0.197.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:33025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:45:23.883311 16670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:45:23.883633 16670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:45:23.883809 16670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:45:23.891975 16670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:45:23.892102 16670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:45:23.892282 16670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.67
I20260929 16:45:23.903180 16670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:33025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab
--principal=kudu/127.0.197.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.197.67
--webserver_port=0
--tserver_master_addrs=127.0.197.126:37141
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.67
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:45:23.904567 16670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:45:23.906044 16670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:45:23.921468 16675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:45:23.921530 16676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:45:23.922343 16678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:45:23.922621 16670 server_base.cc:1034] running on GCE node
I20260929 16:45:23.923394 16670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:45:23.924468 16670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:45:23.925730 16670 hybrid_clock.cc:648] HybridClock initialized: now 1790700323925673 us; error 51 us; skew 500 ppm
I20260929 16:45:23.926088 16670 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:45:23 dist-test-slave-l31f krb5kdc[16296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700323, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:45:23.929129 16530 heartbeater.cc:503] Master 127.0.197.126:37141 was elected leader, sending a full tablet report...
I20260929 16:45:23.929694 16670 init.cc:377] Logged in from keytab as kudu/127.0.197.67@KRBTEST.COM (short username kudu)
I20260929 16:45:23.931633 16670 webserver.cc:460] Webserver started at http://127.0.197.67:39013/ using document root <none> and password file <none>
I20260929 16:45:23.932359 16670 fs_manager.cc:359] Metadata directory not provided
I20260929 16:45:23.932518 16670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:45:23.932839 16670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:45:23.936012 16670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-2/data/instance:
uuid: "ba9242ba774841c0b786fec741f15825"
format_stamp: "Formatted at 2026-09-29 16:45:23 on dist-test-slave-l31f"
server_key: "e6f6b765e1513e73a86fc5be594d7aaf"
server_key_iv: "878dbc3927fc939010fddd3e7521120a"
server_key_version: "encryptionkey@0"
I20260929 16:45:23.936823 16670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-2/wal/instance:
uuid: "ba9242ba774841c0b786fec741f15825"
format_stamp: "Formatted at 2026-09-29 16:45:23 on dist-test-slave-l31f"
server_key: "e6f6b765e1513e73a86fc5be594d7aaf"
server_key_iv: "878dbc3927fc939010fddd3e7521120a"
server_key_version: "encryptionkey@0"
I20260929 16:45:23.942466 16670 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 16:45:23.946372 16685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:45:23.947960 16670 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 16:45:23.948189 16670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-2/wal
uuid: "ba9242ba774841c0b786fec741f15825"
format_stamp: "Formatted at 2026-09-29 16:45:23 on dist-test-slave-l31f"
server_key: "e6f6b765e1513e73a86fc5be594d7aaf"
server_key_iv: "878dbc3927fc939010fddd3e7521120a"
server_key_version: "encryptionkey@0"
I20260929 16:45:23.948424 16670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:45:23.985659 16670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:45:23.990489 16670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:45:23.990833 16670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:45:23.991914 16670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:45:23.993376 16670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:45:23.993492 16670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:45:23.993661 16670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:45:23.993736 16670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:45:24.029780 16670 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.67:39065
I20260929 16:45:24.029837 16797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.67:39065 every 8 connection(s)
I20260929 16:45:24.031234 16670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
I20260929 16:45:24.031903 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 16670
I20260929 16:45:24.032162 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-2/wal/instance
I20260929 16:45:24.032745 789 external_mini_cluster.cc:1428] Setting key ccdc9d4fcb7b14598245ef9473675085
Sep 29 16:45:24 dist-test-slave-l31f krb5kdc[16296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700323, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.67@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:45:24.054152 16800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:24.034263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:35863 (local address 127.0.197.126:37141)
0929 16:45:24.034798 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0929 16:45:24.034805 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:45:24.036169 (+ 1364us) server_negotiation.cc:407] Connection header received
0929 16:45:24.038218 (+ 2049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:45:24.038223 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:45:24.038317 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:45:24.038450 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:45:24.040301 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:24.041055 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:24.042732 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:24.043238 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:24.043386 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:24.047316 (+ 3930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:45:24.047321 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:45:24.047324 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:45:24.047361 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:45:24.050585 (+ 3224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:45:24.051600 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:45:24.051604 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:45:24.051606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:45:24.051752 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:45:24.052368 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:45:24.052371 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:45:24.052374 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:45:24.052666 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:45:24.052789 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 16:45:24.053675 (+ 886us) server_negotiation.cc:299] Negotiation successful
0929 16:45:24.053862 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":263,"threads_started":1}
I20260929 16:45:24.055141 16801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:24.035002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.67:35863)
0929 16:45:24.035995 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:24.036058 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 16:45:24.037910 (+ 1852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:45:24.038727 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:45:24.038736 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:45:24.039459 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:45:24.040089 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:45:24.040111 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:45:24.041251 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:45:24.041257 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:45:24.042540 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:45:24.042554 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:45:24.043500 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:45:24.043508 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:45:24.043761 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:24.044488 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:45:24.044518 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:45:24.046949 (+ 2431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:45:24.050770 (+ 3821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:45:24.050778 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:45:24.050795 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:45:24.051456 (+ 661us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:45:24.051960 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:45:24.051964 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:45:24.051967 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:45:24.052205 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:45:24.052783 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:45:24.052790 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:45:24.053173 (+ 383us) client_negotiation.cc:769] Sending connection context
0929 16:45:24.053435 (+ 262us) client_negotiation.cc:240] Negotiation successful
0929 16:45:24.053915 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":370,"threads_started":1}
I20260929 16:45:24.056478 16798 heartbeater.cc:347] Connected to a master server at 127.0.197.126:37141
I20260929 16:45:24.056769 16798 heartbeater.cc:464] Registering TS with master...
I20260929 16:45:24.057441 16798 heartbeater.cc:511] Master 127.0.197.126:37141 requested a full tablet report, sending...
I20260929 16:45:24.058810 16341 ts_manager.cc:194] Registered new tserver with Master: ba9242ba774841c0b786fec741f15825 (127.0.197.67:39065)
I20260929 16:45:24.059674 16341 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.67@KRBTEST.COM'} at 127.0.197.67:35863
I20260929 16:45:24.061918 789 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:45:24.080953 16800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:24.066572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40800 (local address 127.0.197.126:37141)
0929 16:45:24.066877 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0929 16:45:24.066884 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:45:24.067233 (+ 349us) server_negotiation.cc:407] Connection header received
0929 16:45:24.067482 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:45:24.067492 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:45:24.067572 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:45:24.067685 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:45:24.069172 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:24.070341 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:24.071924 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:24.072353 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:24.072458 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:24.075219 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:45:24.075225 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:45:24.075229 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:45:24.075269 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:45:24.077886 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:45:24.078747 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:45:24.078751 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:45:24.078753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:45:24.078895 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:45:24.079386 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:45:24.079390 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:45:24.079392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:45:24.079764 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:45:24.079973 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0929 16:45:24.080347 (+ 374us) server_negotiation.cc:299] Negotiation successful
0929 16:45:24.080576 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260929 16:45:24.085743 16341 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40800:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:45:24.092944 16341 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:45:24.118498 16813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:24.106448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:41857 (local address 127.0.0.1:59380)
0929 16:45:24.107905 (+ 1457us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:24.107952 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 16:45:24.108154 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:45:24.109248 (+ 1094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:45:24.109256 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:45:24.109950 (+ 694us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:45:24.110405 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:45:24.110419 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:45:24.112484 (+ 2065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:45:24.112489 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:45:24.114890 (+ 2401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:45:24.114905 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:45:24.117383 (+ 2478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:45:24.117388 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:45:24.117622 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:24.117677 (+ 55us) client_negotiation.cc:769] Sending connection context
0929 16:45:24.117824 (+ 147us) client_negotiation.cc:240] Negotiation successful
0929 16:45:24.118035 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1168,"spinlock_wait_cycles":6016,"thread_start_us":1216,"threads_started":1}
I20260929 16:45:24.119580 16812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:24.106602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59380 (local address 127.0.197.65:41857)
0929 16:45:24.107488 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0929 16:45:24.107497 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:45:24.108337 (+ 840us) server_negotiation.cc:407] Connection header received
0929 16:45:24.108456 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:45:24.108466 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:45:24.108761 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:45:24.108952 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:45:24.110628 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:24.112306 (+ 1678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:24.115134 (+ 2828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:24.116802 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:24.116930 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:24.117183 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0929 16:45:24.118999 (+ 1816us) server_negotiation.cc:299] Negotiation successful
0929 16:45:24.119224 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":445,"threads_started":1}
I20260929 16:45:24.122088 16814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:24.107515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42780 (local address 127.0.197.66:43983)
0929 16:45:24.108729 (+ 1214us) server_negotiation.cc:206] Beginning negotiation
0929 16:45:24.108738 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:45:24.108765 (+ 27us) server_negotiation.cc:407] Connection header received
0929 16:45:24.108864 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:45:24.108874 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:45:24.109106 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:45:24.109290 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:45:24.111997 (+ 2707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:24.114902 (+ 2905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:24.119102 (+ 4200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:24.120785 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:24.120943 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:24.121183 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0929 16:45:24.121546 (+ 363us) server_negotiation.cc:299] Negotiation successful
0929 16:45:24.121747 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1059,"thread_start_us":930,"threads_started":1}
I20260929 16:45:24.122131 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:24.106982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.66:43983 (local address 127.0.0.1:42780)
0929 16:45:24.107958 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:24.107977 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:45:24.108154 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:45:24.109535 (+ 1381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:45:24.109541 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:45:24.111241 (+ 1700us) spinlock_profiling.cc:243] Waited 1.15 ms on lock 0x61200002c928. stack: 00007f98a79ff66e 00007f98a79ff430 00007f98a6a865ed 00000000008ea064 00000000008ea02d 00000000008e99fa 00007f98a79f50fd 00007f98a79f4d1e 00007f98ba8fe3cd 00007f98ba8fe36c 00007f98ba8cab3b 00007f98ae4068e9 00007f98ae402e45 00007f98ae48adca 00007f98ae488f68 00007f98ae4bc82f
0929 16:45:24.111267 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:45:24.111757 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:45:24.111774 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:45:24.115179 (+ 3405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:45:24.115185 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:45:24.117279 (+ 2094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:45:24.117297 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:45:24.121105 (+ 3808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:45:24.121111 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:45:24.121310 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:24.121346 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 16:45:24.121438 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 16:45:24.121583 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":851,"spinlock_wait_cycles":2648320,"thread_start_us":705,"threads_started":1}
I20260929 16:45:24.124014 16465 tablet_service.cc:1467] Processing CreateTablet for tablet 95fbca0398734011bac607ae5acebc77 (DEFAULT_TABLE table=test-table [id=ea193c0bf88b4d7bb185493bb98647ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:45:24.125671 16465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95fbca0398734011bac607ae5acebc77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:45:24.126484 16599 tablet_service.cc:1467] Processing CreateTablet for tablet 95fbca0398734011bac607ae5acebc77 (DEFAULT_TABLE table=test-table [id=ea193c0bf88b4d7bb185493bb98647ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:45:24.127995 16599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95fbca0398734011bac607ae5acebc77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:45:24.128577 16815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:24.108527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.67:39065 (local address 127.0.0.1:36684)
0929 16:45:24.120057 (+ 11530us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:24.120084 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 16:45:24.120254 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:45:24.121859 (+ 1605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:45:24.121864 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:45:24.121894 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:45:24.122320 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:45:24.122333 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:45:24.124420 (+ 2087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:45:24.124425 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:45:24.126158 (+ 1733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:45:24.126173 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:45:24.127628 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:45:24.127634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:45:24.127809 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:24.127851 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 16:45:24.127981 (+ 130us) client_negotiation.cc:240] Negotiation successful
0929 16:45:24.128201 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11339,"thread_start_us":11845,"threads_started":1}
I20260929 16:45:24.128577 16816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:24.109152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36684 (local address 127.0.197.67:39065)
0929 16:45:24.121132 (+ 11980us) server_negotiation.cc:206] Beginning negotiation
0929 16:45:24.121140 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:45:24.121174 (+ 34us) server_negotiation.cc:407] Connection header received
0929 16:45:24.121273 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:45:24.121282 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:45:24.121568 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:45:24.122537 (+ 969us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:45:24.122841 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:24.124267 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:24.126326 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:24.127414 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:24.127572 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:24.127808 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0929 16:45:24.128028 (+ 220us) server_negotiation.cc:299] Negotiation successful
0929 16:45:24.128199 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11801,"thread_start_us":11645,"threads_started":1}
I20260929 16:45:24.132593 16733 tablet_service.cc:1467] Processing CreateTablet for tablet 95fbca0398734011bac607ae5acebc77 (DEFAULT_TABLE table=test-table [id=ea193c0bf88b4d7bb185493bb98647ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:45:24.133950 16733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95fbca0398734011bac607ae5acebc77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:45:24.141639 16817 tablet_bootstrap.cc:492] T 95fbca0398734011bac607ae5acebc77 P a1e2ec51ef5d43cbb18df0696c0d159d: Bootstrap starting.
I20260929 16:45:24.143966 16818 tablet_bootstrap.cc:492] T 95fbca0398734011bac607ae5acebc77 P ddc41210f2dc4a3189be59eea46a2e31: Bootstrap starting.
I20260929 16:45:24.145854 16817 tablet_bootstrap.cc:654] T 95fbca0398734011bac607ae5acebc77 P a1e2ec51ef5d43cbb18df0696c0d159d: Neither blocks nor log segments found. Creating new log.
I20260929 16:45:24.147609 16818 tablet_bootstrap.cc:654] T 95fbca0398734011bac607ae5acebc77 P ddc41210f2dc4a3189be59eea46a2e31: Neither blocks nor log segments found. Creating new log.
I20260929 16:45:24.147545 16817 log.cc:824] T 95fbca0398734011bac607ae5acebc77 P a1e2ec51ef5d43cbb18df0696c0d159d: Log is configured to *not* fsync() on all Append() calls
I20260929 16:45:24.148986 16818 log.cc:824] T 95fbca0398734011bac607ae5acebc77 P ddc41210f2dc4a3189be59eea46a2e31: Log is configured to *not* fsync() on all Append() calls
I20260929 16:45:24.150123 16819 tablet_bootstrap.cc:492] T 95fbca0398734011bac607ae5acebc77 P ba9242ba774841c0b786fec741f15825: Bootstrap starting.
I20260929 16:45:24.151464 16817 tablet_bootstrap.cc:492] T 95fbca0398734011bac607ae5acebc77 P a1e2ec51ef5d43cbb18df0696c0d159d: No bootstrap required, opened a new log
I20260929 16:45:24.151793 16817 ts_tablet_manager.cc:1397] T 95fbca0398734011bac607ae5acebc77 P a1e2ec51ef5d43cbb18df0696c0d159d: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.002s
I20260929 16:45:24.152103 16818 tablet_bootstrap.cc:492] T 95fbca0398734011bac607ae5acebc77 P ddc41210f2dc4a3189be59eea46a2e31: No bootstrap required, opened a new log
I20260929 16:45:24.152444 16818 ts_tablet_manager.cc:1397] T 95fbca0398734011bac607ae5acebc77 P ddc41210f2dc4a3189be59eea46a2e31: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 16:45:24.154129 16819 tablet_bootstrap.cc:654] T 95fbca0398734011bac607ae5acebc77 P ba9242ba774841c0b786fec741f15825: Neither blocks nor log segments found. Creating new log.
I20260929 16:45:24.155575 16819 log.cc:824] T 95fbca0398734011bac607ae5acebc77 P ba9242ba774841c0b786fec741f15825: Log is configured to *not* fsync() on all Append() calls
I20260929 16:45:24.159286 16819 tablet_bootstrap.cc:492] T 95fbca0398734011bac607ae5acebc77 P ba9242ba774841c0b786fec741f15825: No bootstrap required, opened a new log
I20260929 16:45:24.159622 16819 ts_tablet_manager.cc:1397] T 95fbca0398734011bac607ae5acebc77 P ba9242ba774841c0b786fec741f15825: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260929 16:45:24.161545 16818 raft_consensus.cc:348] T 95fbca0398734011bac607ae5acebc77 P ddc41210f2dc4a3189be59eea46a2e31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e2ec51ef5d43cbb18df0696c0d159d" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 41857 } } peers { permanent_uuid: "ddc41210f2dc4a3189be59eea46a2e31" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43983 } } peers { permanent_uuid: "ba9242ba774841c0b786fec741f15825" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 39065 } }
I20260929 16:45:24.161983 16818 raft_consensus.cc:374] T 95fbca0398734011bac607ae5acebc77 P ddc41210f2dc4a3189be59eea46a2e31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:45:24.162089 16818 raft_consensus.cc:729] T 95fbca0398734011bac607ae5acebc77 P ddc41210f2dc4a3189be59eea46a2e31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddc41210f2dc4a3189be59eea46a2e31, State: Initialized, Role: FOLLOWER
I20260929 16:45:24.162897 16818 consensus_queue.cc:260] T 95fbca0398734011bac607ae5acebc77 P ddc41210f2dc4a3189be59eea46a2e31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e2ec51ef5d43cbb18df0696c0d159d" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 41857 } } peers { permanent_uuid: "ddc41210f2dc4a3189be59eea46a2e31" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43983 } } peers { permanent_uuid: "ba9242ba774841c0b786fec741f15825" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 39065 } }
I20260929 16:45:24.169371 16817 raft_consensus.cc:348] T 95fbca0398734011bac607ae5acebc77 P a1e2ec51ef5d43cbb18df0696c0d159d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e2ec51ef5d43cbb18df0696c0d159d" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 41857 } } peers { permanent_uuid: "ddc41210f2dc4a3189be59eea46a2e31" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43983 } } peers { permanent_uuid: "ba9242ba774841c0b786fec741f15825" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 39065 } }
I20260929 16:45:24.170217 16817 raft_consensus.cc:374] T 95fbca0398734011bac607ae5acebc77 P a1e2ec51ef5d43cbb18df0696c0d159d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:45:24.170397 16817 raft_consensus.cc:729] T 95fbca0398734011bac607ae5acebc77 P a1e2ec51ef5d43cbb18df0696c0d159d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1e2ec51ef5d43cbb18df0696c0d159d, State: Initialized, Role: FOLLOWER
I20260929 16:45:24.171334 16819 raft_consensus.cc:348] T 95fbca0398734011bac607ae5acebc77 P ba9242ba774841c0b786fec741f15825 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e2ec51ef5d43cbb18df0696c0d159d" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 41857 } } peers { permanent_uuid: "ddc41210f2dc4a3189be59eea46a2e31" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43983 } } peers { permanent_uuid: "ba9242ba774841c0b786fec741f15825" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 39065 } }
I20260929 16:45:24.171779 16819 raft_consensus.cc:374] T 95fbca0398734011bac607ae5acebc77 P ba9242ba774841c0b786fec741f15825 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:45:24.171875 16819 raft_consensus.cc:729] T 95fbca0398734011bac607ae5acebc77 P ba9242ba774841c0b786fec741f15825 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba9242ba774841c0b786fec741f15825, State: Initialized, Role: FOLLOWER
I20260929 16:45:24.172612 16819 consensus_queue.cc:260] T 95fbca0398734011bac607ae5acebc77 P ba9242ba774841c0b786fec741f15825 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e2ec51ef5d43cbb18df0696c0d159d" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 41857 } } peers { permanent_uuid: "ddc41210f2dc4a3189be59eea46a2e31" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43983 } } peers { permanent_uuid: "ba9242ba774841c0b786fec741f15825" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 39065 } }
I20260929 16:45:24.173610 16817 consensus_queue.cc:260] T 95fbca0398734011bac607ae5acebc77 P a1e2ec51ef5d43cbb18df0696c0d159d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e2ec51ef5d43cbb18df0696c0d159d" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 41857 } } peers { permanent_uuid: "ddc41210f2dc4a3189be59eea46a2e31" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43983 } } peers { permanent_uuid: "ba9242ba774841c0b786fec741f15825" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 39065 } }
I20260929 16:45:24.179981 16664 heartbeater.cc:503] Master 127.0.197.126:37141 was elected leader, sending a full tablet report...
I20260929 16:45:24.180924 16819 ts_tablet_manager.cc:1428] T 95fbca0398734011bac607ae5acebc77 P ba9242ba774841c0b786fec741f15825: Time spent starting tablet: real 0.021s user 0.016s sys 0.004s
I20260929 16:45:24.181089 16818 ts_tablet_manager.cc:1428] T 95fbca0398734011bac607ae5acebc77 P ddc41210f2dc4a3189be59eea46a2e31: Time spent starting tablet: real 0.029s user 0.018s sys 0.004s
I20260929 16:45:24.181349 16798 heartbeater.cc:503] Master 127.0.197.126:37141 was elected leader, sending a full tablet report...
I20260929 16:45:24.181646 16823 raft_consensus.cc:482] T 95fbca0398734011bac607ae5acebc77 P ddc41210f2dc4a3189be59eea46a2e31 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:45:24.181939 16823 raft_consensus.cc:504] T 95fbca0398734011bac607ae5acebc77 P ddc41210f2dc4a3189be59eea46a2e31 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e2ec51ef5d43cbb18df0696c0d159d" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 41857 } } peers { permanent_uuid: "ddc41210f2dc4a3189be59eea46a2e31" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43983 } } peers { permanent_uuid: "ba9242ba774841c0b786fec741f15825" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 39065 } }
I20260929 16:45:24.195398 16823 leader_election.cc:288] T 95fbca0398734011bac607ae5acebc77 P ddc41210f2dc4a3189be59eea46a2e31 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1e2ec51ef5d43cbb18df0696c0d159d (127.0.197.65:41857), ba9242ba774841c0b786fec741f15825 (127.0.197.67:39065)
I20260929 16:45:24.199221 16817 ts_tablet_manager.cc:1428] T 95fbca0398734011bac607ae5acebc77 P a1e2ec51ef5d43cbb18df0696c0d159d: Time spent starting tablet: real 0.047s user 0.024s sys 0.015s
I20260929 16:45:24.204556 16816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:24.185056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:42645 (local address 127.0.197.67:39065)
0929 16:45:24.185263 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 16:45:24.185269 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:45:24.196373 (+ 11104us) server_negotiation.cc:407] Connection header received
0929 16:45:24.196672 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:45:24.196677 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:45:24.196752 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:45:24.196813 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:45:24.197736 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:24.198859 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:24.201768 (+ 2909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:24.203130 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:24.203276 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:24.203372 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 16:45:24.204132 (+ 760us) server_negotiation.cc:299] Negotiation successful
0929 16:45:24.204295 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 16:45:24.204662 16827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:24.184546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.67:39065 (local address 127.0.197.66:42645)
0929 16:45:24.196244 (+ 11698us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:24.196267 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 16:45:24.196513 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:45:24.197051 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:45:24.197056 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:45:24.197088 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:45:24.197531 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:45:24.197544 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:45:24.199100 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:45:24.199105 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:45:24.201548 (+ 2443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:45:24.201566 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:45:24.203639 (+ 2073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:45:24.203646 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:45:24.203856 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:24.203894 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 16:45:24.204060 (+ 166us) client_negotiation.cc:240] Negotiation successful
0929 16:45:24.204225 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11524,"thread_start_us":11400,"threads_started":1}
I20260929 16:45:24.205892 16753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95fbca0398734011bac607ae5acebc77" candidate_uuid: "ddc41210f2dc4a3189be59eea46a2e31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba9242ba774841c0b786fec741f15825" is_pre_election: true
I20260929 16:45:24.206583 16753 raft_consensus.cc:2454] T 95fbca0398734011bac607ae5acebc77 P ba9242ba774841c0b786fec741f15825 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddc41210f2dc4a3189be59eea46a2e31 in term 0.
I20260929 16:45:24.207394 16553 leader_election.cc:302] T 95fbca0398734011bac607ae5acebc77 P ddc41210f2dc4a3189be59eea46a2e31 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba9242ba774841c0b786fec741f15825, ddc41210f2dc4a3189be59eea46a2e31; no voters:
I20260929 16:45:24.207918 16823 raft_consensus.cc:2781] T 95fbca0398734011bac607ae5acebc77 P ddc41210f2dc4a3189be59eea46a2e31 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:45:24.208096 16823 raft_consensus.cc:482] T 95fbca0398734011bac607ae5acebc77 P ddc41210f2dc4a3189be59eea46a2e31 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:45:24.208211 16823 raft_consensus.cc:3037] T 95fbca0398734011bac607ae5acebc77 P ddc41210f2dc4a3189be59eea46a2e31 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:45:24.209246 16812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:24.198083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:55887 (local address 127.0.197.65:41857)
0929 16:45:24.198294 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 16:45:24.198300 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:45:24.200649 (+ 2349us) server_negotiation.cc:407] Connection header received
0929 16:45:24.200942 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:45:24.200948 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:45:24.201023 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:45:24.201101 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:45:24.201969 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:24.203099 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:24.206635 (+ 3536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:24.207975 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:24.208178 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:24.208258 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 16:45:24.208809 (+ 551us) server_negotiation.cc:299] Negotiation successful
0929 16:45:24.208956 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 16:45:24.209347 16828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:24.198103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:41857 (local address 127.0.197.66:55887)
0929 16:45:24.200519 (+ 2416us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:24.200548 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 16:45:24.200763 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:45:24.201303 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:45:24.201307 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:45:24.201333 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:45:24.201778 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:45:24.201795 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:45:24.203290 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:45:24.203295 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:45:24.206374 (+ 3079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:45:24.206395 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:45:24.208444 (+ 2049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:45:24.208452 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:45:24.208621 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:24.208648 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 16:45:24.208777 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 16:45:24.208942 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2055,"thread_start_us":1769,"threads_started":1}
I20260929 16:45:24.210867 16485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95fbca0398734011bac607ae5acebc77" candidate_uuid: "ddc41210f2dc4a3189be59eea46a2e31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1e2ec51ef5d43cbb18df0696c0d159d" is_pre_election: true
W20260929 16:45:24.211438 16665 tablet.cc:2369] T 95fbca0398734011bac607ae5acebc77 P ddc41210f2dc4a3189be59eea46a2e31: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:45:24.211597 16485 raft_consensus.cc:2454] T 95fbca0398734011bac607ae5acebc77 P a1e2ec51ef5d43cbb18df0696c0d159d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddc41210f2dc4a3189be59eea46a2e31 in term 0.
I20260929 16:45:24.212405 16823 raft_consensus.cc:504] T 95fbca0398734011bac607ae5acebc77 P ddc41210f2dc4a3189be59eea46a2e31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e2ec51ef5d43cbb18df0696c0d159d" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 41857 } } peers { permanent_uuid: "ddc41210f2dc4a3189be59eea46a2e31" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43983 } } peers { permanent_uuid: "ba9242ba774841c0b786fec741f15825" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 39065 } }
I20260929 16:45:24.213251 16823 leader_election.cc:288] T 95fbca0398734011bac607ae5acebc77 P ddc41210f2dc4a3189be59eea46a2e31 [CANDIDATE]: Term 1 election: Requested vote from peers a1e2ec51ef5d43cbb18df0696c0d159d (127.0.197.65:41857), ba9242ba774841c0b786fec741f15825 (127.0.197.67:39065)
I20260929 16:45:24.213822 16485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95fbca0398734011bac607ae5acebc77" candidate_uuid: "ddc41210f2dc4a3189be59eea46a2e31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1e2ec51ef5d43cbb18df0696c0d159d"
I20260929 16:45:24.213877 16753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95fbca0398734011bac607ae5acebc77" candidate_uuid: "ddc41210f2dc4a3189be59eea46a2e31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba9242ba774841c0b786fec741f15825"
I20260929 16:45:24.214082 16485 raft_consensus.cc:3037] T 95fbca0398734011bac607ae5acebc77 P a1e2ec51ef5d43cbb18df0696c0d159d [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:45:24.214165 16753 raft_consensus.cc:3037] T 95fbca0398734011bac607ae5acebc77 P ba9242ba774841c0b786fec741f15825 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:45:24.216614 16485 raft_consensus.cc:2454] T 95fbca0398734011bac607ae5acebc77 P a1e2ec51ef5d43cbb18df0696c0d159d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddc41210f2dc4a3189be59eea46a2e31 in term 1.
I20260929 16:45:24.216995 16753 raft_consensus.cc:2454] T 95fbca0398734011bac607ae5acebc77 P ba9242ba774841c0b786fec741f15825 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddc41210f2dc4a3189be59eea46a2e31 in term 1.
I20260929 16:45:24.217309 16555 leader_election.cc:302] T 95fbca0398734011bac607ae5acebc77 P ddc41210f2dc4a3189be59eea46a2e31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1e2ec51ef5d43cbb18df0696c0d159d, ddc41210f2dc4a3189be59eea46a2e31; no voters:
I20260929 16:45:24.217789 16823 raft_consensus.cc:2781] T 95fbca0398734011bac607ae5acebc77 P ddc41210f2dc4a3189be59eea46a2e31 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:45:24.218001 16823 raft_consensus.cc:686] T 95fbca0398734011bac607ae5acebc77 P ddc41210f2dc4a3189be59eea46a2e31 [term 1 LEADER]: Becoming Leader. State: Replica: ddc41210f2dc4a3189be59eea46a2e31, State: Running, Role: LEADER
I20260929 16:45:24.218585 16823 consensus_queue.cc:237] T 95fbca0398734011bac607ae5acebc77 P ddc41210f2dc4a3189be59eea46a2e31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e2ec51ef5d43cbb18df0696c0d159d" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 41857 } } peers { permanent_uuid: "ddc41210f2dc4a3189be59eea46a2e31" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43983 } } peers { permanent_uuid: "ba9242ba774841c0b786fec741f15825" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 39065 } }
I20260929 16:45:24.225028 16341 catalog_manager.cc:5495] T 95fbca0398734011bac607ae5acebc77 P ddc41210f2dc4a3189be59eea46a2e31 reported cstate change: term changed from 0 to 1, leader changed from <none> to ddc41210f2dc4a3189be59eea46a2e31 (127.0.197.66). New cstate: current_term: 1 leader_uuid: "ddc41210f2dc4a3189be59eea46a2e31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e2ec51ef5d43cbb18df0696c0d159d" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 41857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddc41210f2dc4a3189be59eea46a2e31" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba9242ba774841c0b786fec741f15825" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 39065 } health_report { overall_health: UNKNOWN } } }
I20260929 16:45:24.247247 16814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:24.241414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52572 (local address 127.0.197.66:43983)
0929 16:45:24.241637 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 16:45:24.241644 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:45:24.241670 (+ 26us) server_negotiation.cc:407] Connection header received
0929 16:45:24.241804 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:45:24.241808 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:45:24.241892 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:45:24.242075 (+ 183us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:45:24.242504 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:24.243316 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:24.244877 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:24.245306 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:24.245462 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:24.246003 (+ 541us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:45:24.246559 (+ 556us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:45:24.246723 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 16:45:24.246828 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 16:45:24.246919 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 16:45:24.256407 16753 raft_consensus.cc:1261] T 95fbca0398734011bac607ae5acebc77 P ba9242ba774841c0b786fec741f15825 [term 1 FOLLOWER]: Refusing update from remote peer ddc41210f2dc4a3189be59eea46a2e31: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:45:24.256407 16485 raft_consensus.cc:1261] T 95fbca0398734011bac607ae5acebc77 P a1e2ec51ef5d43cbb18df0696c0d159d [term 1 FOLLOWER]: Refusing update from remote peer ddc41210f2dc4a3189be59eea46a2e31: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:45:24.257531 16824 consensus_queue.cc:1035] T 95fbca0398734011bac607ae5acebc77 P ddc41210f2dc4a3189be59eea46a2e31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1e2ec51ef5d43cbb18df0696c0d159d" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 41857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:45:24.257905 16823 consensus_queue.cc:1035] T 95fbca0398734011bac607ae5acebc77 P ddc41210f2dc4a3189be59eea46a2e31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba9242ba774841c0b786fec741f15825" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 39065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:45:24.269397 16831 mvcc.cc:205] Tried to move back new op lower bound from 7334708528143302656 to 7334708528003981312. Current Snapshot: MvccSnapshot[applied={T|T < 7334708528143302656}]
I20260929 16:45:24.291208 16340 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:45:24.291539 16340 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:45:24.295984 16340 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e: Sending DeleteTablet for 3 replicas of tablet 95fbca0398734011bac607ae5acebc77
I20260929 16:45:24.297431 16465 tablet_service.cc:1514] Processing DeleteTablet for tablet 95fbca0398734011bac607ae5acebc77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:45:24 UTC) from {username='kudu', principal='kudu/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:59380
I20260929 16:45:24.297444 16733 tablet_service.cc:1514] Processing DeleteTablet for tablet 95fbca0398734011bac607ae5acebc77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:45:24 UTC) from {username='kudu', principal='kudu/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:36684
I20260929 16:45:24.297538 16599 tablet_service.cc:1514] Processing DeleteTablet for tablet 95fbca0398734011bac607ae5acebc77 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:45:24 UTC) from {username='kudu', principal='kudu/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:42780
I20260929 16:45:24.299201 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 16402
I20260929 16:45:24.313109 16841 tablet_replica.cc:330] T 95fbca0398734011bac607ae5acebc77 P ba9242ba774841c0b786fec741f15825: stopping tablet replica
I20260929 16:45:24.313752 16841 raft_consensus.cc:2229] T 95fbca0398734011bac607ae5acebc77 P ba9242ba774841c0b786fec741f15825 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:45:24.314405 16841 raft_consensus.cc:2258] T 95fbca0398734011bac607ae5acebc77 P ba9242ba774841c0b786fec741f15825 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:45:24.317240 16841 ts_tablet_manager.cc:1905] T 95fbca0398734011bac607ae5acebc77 P ba9242ba774841c0b786fec741f15825: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:45:24.324645 16840 tablet_replica.cc:330] T 95fbca0398734011bac607ae5acebc77 P a1e2ec51ef5d43cbb18df0696c0d159d: stopping tablet replica
I20260929 16:45:24.326193 16841 ts_tablet_manager.cc:1918] T 95fbca0398734011bac607ae5acebc77 P ba9242ba774841c0b786fec741f15825: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:45:24.326395 16841 log.cc:1196] T 95fbca0398734011bac607ae5acebc77 P ba9242ba774841c0b786fec741f15825: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-2/wal/wals/95fbca0398734011bac607ae5acebc77
I20260929 16:45:24.326944 16841 ts_tablet_manager.cc:1939] T 95fbca0398734011bac607ae5acebc77 P ba9242ba774841c0b786fec741f15825: Deleting consensus metadata
I20260929 16:45:24.327158 16840 raft_consensus.cc:2229] T 95fbca0398734011bac607ae5acebc77 P a1e2ec51ef5d43cbb18df0696c0d159d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:45:24.328414 16840 raft_consensus.cc:2258] T 95fbca0398734011bac607ae5acebc77 P a1e2ec51ef5d43cbb18df0696c0d159d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:45:24.329087 16326 catalog_manager.cc:4841] TS ba9242ba774841c0b786fec741f15825 (127.0.197.67:39065): tablet 95fbca0398734011bac607ae5acebc77 (table test-table [id=ea193c0bf88b4d7bb185493bb98647ea]) successfully deleted
I20260929 16:45:24.332381 16842 tablet_replica.cc:330] T 95fbca0398734011bac607ae5acebc77 P ddc41210f2dc4a3189be59eea46a2e31: stopping tablet replica
I20260929 16:45:24.333057 16842 raft_consensus.cc:2229] T 95fbca0398734011bac607ae5acebc77 P ddc41210f2dc4a3189be59eea46a2e31 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:45:24.334064 16842 raft_consensus.cc:2258] T 95fbca0398734011bac607ae5acebc77 P ddc41210f2dc4a3189be59eea46a2e31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:45:24.334941 16840 ts_tablet_manager.cc:1905] T 95fbca0398734011bac607ae5acebc77 P a1e2ec51ef5d43cbb18df0696c0d159d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:45:24.336632 16842 ts_tablet_manager.cc:1905] T 95fbca0398734011bac607ae5acebc77 P ddc41210f2dc4a3189be59eea46a2e31: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 16:45:24 dist-test-slave-l31f krb5kdc[16296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.65@KRBTEST.COM
I20260929 16:45:24.342729 16840 ts_tablet_manager.cc:1918] T 95fbca0398734011bac607ae5acebc77 P a1e2ec51ef5d43cbb18df0696c0d159d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:45:24.342955 16840 log.cc:1196] T 95fbca0398734011bac607ae5acebc77 P a1e2ec51ef5d43cbb18df0696c0d159d: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-0/wal/wals/95fbca0398734011bac607ae5acebc77
I20260929 16:45:24.343540 16840 ts_tablet_manager.cc:1939] T 95fbca0398734011bac607ae5acebc77 P a1e2ec51ef5d43cbb18df0696c0d159d: Deleting consensus metadata
I20260929 16:45:24.345715 16328 catalog_manager.cc:4841] TS a1e2ec51ef5d43cbb18df0696c0d159d (127.0.197.65:41857): tablet 95fbca0398734011bac607ae5acebc77 (table test-table [id=ea193c0bf88b4d7bb185493bb98647ea]) successfully deleted
I20260929 16:45:24.346534 16842 ts_tablet_manager.cc:1918] T 95fbca0398734011bac607ae5acebc77 P ddc41210f2dc4a3189be59eea46a2e31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:45:24.346781 16842 log.cc:1196] T 95fbca0398734011bac607ae5acebc77 P ddc41210f2dc4a3189be59eea46a2e31: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-1/wal/wals/95fbca0398734011bac607ae5acebc77
I20260929 16:45:24.347543 16842 ts_tablet_manager.cc:1939] T 95fbca0398734011bac607ae5acebc77 P ddc41210f2dc4a3189be59eea46a2e31: Deleting consensus metadata
I20260929 16:45:24.349290 16812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:24.300782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48326 (local address 127.0.197.65:41857)
0929 16:45:24.301060 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0929 16:45:24.301069 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:45:24.323708 (+ 22639us) server_negotiation.cc:407] Connection header received
0929 16:45:24.323955 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:45:24.323963 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:45:24.324099 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:45:24.324252 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:45:24.328694 (+ 4442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:24.329749 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:24.331398 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:24.332142 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:24.332301 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:24.339290 (+ 6989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:45:24.339305 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:45:24.339324 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:45:24.339386 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:45:24.344561 (+ 5175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:45:24.345893 (+ 1332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:45:24.345906 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:45:24.345910 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:45:24.346133 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:45:24.346779 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:45:24.346785 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:45:24.346788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:45:24.347421 (+ 633us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:45:24.347804 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0929 16:45:24.348606 (+ 802us) server_negotiation.cc:299] Negotiation successful
0929 16:45:24.348889 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 16:45:24.349929 16326 catalog_manager.cc:4841] TS ddc41210f2dc4a3189be59eea46a2e31 (127.0.197.66:43983): tablet 95fbca0398734011bac607ae5acebc77 (table test-table [id=ea193c0bf88b4d7bb185493bb98647ea]) successfully deleted
I20260929 16:45:24.350363 16525 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:45:24.448635 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 16402
I20260929 16:45:24.474819 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 16536
Sep 29 16:45:24 dist-test-slave-l31f krb5kdc[16296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.66@KRBTEST.COM
I20260929 16:45:24.495543 16814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:24.475676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52588 (local address 127.0.197.66:43983)
0929 16:45:24.476271 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0929 16:45:24.476281 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:45:24.476299 (+ 18us) server_negotiation.cc:407] Connection header received
0929 16:45:24.476359 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:45:24.476364 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:45:24.476452 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:45:24.476575 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:45:24.478673 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:24.480150 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:24.482308 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:24.482801 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:24.482966 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:24.488474 (+ 5508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:45:24.488492 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:45:24.488508 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:45:24.488564 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:45:24.492104 (+ 3540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:45:24.493001 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:45:24.493009 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:45:24.493012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:45:24.493204 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:45:24.493646 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:45:24.493652 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:45:24.493654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:45:24.494085 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:45:24.494450 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0929 16:45:24.494820 (+ 370us) server_negotiation.cc:299] Negotiation successful
0929 16:45:24.495118 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465}
I20260929 16:45:24.496325 16659 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:45:24.613844 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 16536
I20260929 16:45:24.633222 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 16670
Sep 29 16:45:24 dist-test-slave-l31f krb5kdc[16296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.67@KRBTEST.COM
I20260929 16:45:24.650352 16816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:24.634045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39520 (local address 127.0.197.67:39065)
0929 16:45:24.634237 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 16:45:24.634245 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:45:24.634346 (+ 101us) server_negotiation.cc:407] Connection header received
0929 16:45:24.634579 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:45:24.634587 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:45:24.634675 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:45:24.634798 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:45:24.636293 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:24.637036 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:24.638391 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:24.638810 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:24.638954 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:24.644054 (+ 5100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:45:24.644074 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:45:24.644092 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:45:24.644133 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:45:24.647284 (+ 3151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:45:24.648107 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:45:24.648126 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:45:24.648128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:45:24.648270 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:45:24.648689 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:45:24.648695 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:45:24.648698 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:45:24.649095 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:45:24.649424 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0929 16:45:24.649752 (+ 328us) server_negotiation.cc:299] Negotiation successful
0929 16:45:24.649996 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 16:45:24.651189 16793 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:45:24.743805 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 16670
I20260929 16:45:24.763768 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 16309
I20260929 16:45:24.765121 16371 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:45:24.881177 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 16309
I20260929 16:45:44.904704 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:37141
--webserver_interface=127.0.197.126
--webserver_port=35897
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:33025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.126:37141
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:45:45.274528 16848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:45:45.274955 16848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:45:45.275137 16848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:45:45.282805 16848 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 16:45:45.283041 16848 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 16:45:45.283149 16848 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 16:45:45.283241 16848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:45:45.283344 16848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:45:45.283470 16848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:45:45.283561 16848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:45:45.283643 16848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:45:45.294164 16848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:33025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.197.126:37141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:37141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.197.126
--webserver_port=35897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:45:45.295698 16848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:45:45.297077 16848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:45:45.331691 16854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:45:45.331736 16855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:45:45.331934 16857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:45:45.331741 16848 server_base.cc:1034] running on GCE node
I20260929 16:45:45.333233 16848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:45:45.335176 16848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:45:45.336493 16848 hybrid_clock.cc:648] HybridClock initialized: now 1790700345336418 us; error 50 us; skew 500 ppm
I20260929 16:45:45.336949 16848 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:45:45 dist-test-slave-l31f krb5kdc[16296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:45:45.346483 16848 init.cc:377] Logged in from keytab as kudu/127.0.197.126@KRBTEST.COM (short username kudu)
I20260929 16:45:45.348426 16848 webserver.cc:460] Webserver started at http://127.0.197.126:35897/ using document root <none> and password file <none>
I20260929 16:45:45.349345 16848 fs_manager.cc:359] Metadata directory not provided
I20260929 16:45:45.349521 16848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:45:45.356315 16848 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:45:45.359838 16864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:45:45.361595 16848 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:45:45.361862 16848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "5014d3abe04b487cab15866468715f4e"
format_stamp: "Formatted at 2026-09-29 16:45:22 on dist-test-slave-l31f"
server_key: "37b457cf171f0a8a8f28d7261010da4e"
server_key_iv: "996c493fa1f8594a13099f8a85ac83a5"
server_key_version: "encryptionkey@0"
I20260929 16:45:45.363106 16848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:45:45.387606 16848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:45:45.393100 16848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:45:45.393455 16848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:45:45.414585 16848 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.126:37141
I20260929 16:45:45.414615 16915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.126:37141 every 8 connection(s)
I20260929 16:45:45.416383 16848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:45:45.423352 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 16848
I20260929 16:45:45.424439 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.65:41857
--local_ip_for_outbound_sockets=127.0.197.65
--tserver_master_addrs=127.0.197.126:37141
--webserver_port=43127
--webserver_interface=127.0.197.65
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab
--principal=kudu/127.0.197.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:33025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:45:45.424960 16916 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:45:45.430804 16916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e: Bootstrap starting.
I20260929 16:45:45.455621 16916 log.cc:824] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e: Log is configured to *not* fsync() on all Append() calls
I20260929 16:45:45.471221 16916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:45:45.471823 16916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e: Bootstrap complete.
I20260929 16:45:45.482044 16916 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5014d3abe04b487cab15866468715f4e" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 37141 } }
I20260929 16:45:45.483355 16916 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5014d3abe04b487cab15866468715f4e, State: Initialized, Role: FOLLOWER
I20260929 16:45:45.484396 16916 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5014d3abe04b487cab15866468715f4e" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 37141 } }
I20260929 16:45:45.484753 16916 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:45:45.484895 16916 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:45:45.485106 16916 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e [term 1 FOLLOWER]: Advancing to term 2
I20260929 16:45:45.489645 16916 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5014d3abe04b487cab15866468715f4e" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 37141 } }
I20260929 16:45:45.490726 16916 leader_election.cc:302] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5014d3abe04b487cab15866468715f4e; no voters:
I20260929 16:45:45.491884 16916 leader_election.cc:288] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 16:45:45.492075 16920 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e [term 2 FOLLOWER]: Leader election won for term 2
I20260929 16:45:45.493541 16920 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e [term 2 LEADER]: Becoming Leader. State: Replica: 5014d3abe04b487cab15866468715f4e, State: Running, Role: LEADER
I20260929 16:45:45.494182 16920 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5014d3abe04b487cab15866468715f4e" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 37141 } }
I20260929 16:45:45.494681 16916 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:45:45.497164 16921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5014d3abe04b487cab15866468715f4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5014d3abe04b487cab15866468715f4e" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 37141 } } }
I20260929 16:45:45.497689 16921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e [sys.catalog]: This master's current role is: LEADER
I20260929 16:45:45.501688 16922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5014d3abe04b487cab15866468715f4e. Latest consensus state: current_term: 2 leader_uuid: "5014d3abe04b487cab15866468715f4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5014d3abe04b487cab15866468715f4e" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 37141 } } }
I20260929 16:45:45.502193 16922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e [sys.catalog]: This master's current role is: LEADER
I20260929 16:45:45.504215 16926 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:45:45.515767 16926 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:45:45.517891 16926 catalog_manager.cc:1259] Loaded cluster ID: d7396f1a6df84581bbb913b096f068a6
I20260929 16:45:45.518066 16926 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:45:45.523615 16926 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:45:45.525835 16926 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e: Loaded TSK: 0
I20260929 16:45:45.530939 16937 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 16:45:45.560333 16926 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e: Generated new TSK 1
I20260929 16:45:45.560815 16926 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 16:45:45.831368 16918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:45:45.831736 16918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:45:45.831907 16918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:45:45.840670 16918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:45:45.840817 16918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:45:45.841023 16918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.65
I20260929 16:45:45.852257 16918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:33025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab
--principal=kudu/127.0.197.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.65:41857
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.197.65
--webserver_port=43127
--tserver_master_addrs=127.0.197.126:37141
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.65
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:45:45.853820 16918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:45:45.855412 16918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:45:45.873838 16943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:45:45.873726 16944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:45:45.873701 16946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:45:45.875205 16918 server_base.cc:1034] running on GCE node
I20260929 16:45:45.876086 16918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:45:45.877270 16918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:45:45.878571 16918 hybrid_clock.cc:648] HybridClock initialized: now 1790700345878498 us; error 71 us; skew 500 ppm
I20260929 16:45:45.878984 16918 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:45:45 dist-test-slave-l31f krb5kdc[16296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:45:45.883771 16918 init.cc:377] Logged in from keytab as kudu/127.0.197.65@KRBTEST.COM (short username kudu)
I20260929 16:45:45.885783 16918 webserver.cc:460] Webserver started at http://127.0.197.65:43127/ using document root <none> and password file <none>
I20260929 16:45:45.886605 16918 fs_manager.cc:359] Metadata directory not provided
I20260929 16:45:45.886794 16918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:45:45.893047 16918 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:45:45.896361 16953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:45:45.898258 16918 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 16:45:45.898468 16918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-0/wal
uuid: "a1e2ec51ef5d43cbb18df0696c0d159d"
format_stamp: "Formatted at 2026-09-29 16:45:22 on dist-test-slave-l31f"
server_key: "6cc5d59ebdd86d718375e75d3ed52042"
server_key_iv: "33506d4e02c0d8aced7de2559addef2a"
server_key_version: "encryptionkey@0"
I20260929 16:45:45.899636 16918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:45:45.918711 16918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:45:45.923439 16918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:45:45.923743 16918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:45:45.924800 16918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:45:45.926329 16918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:45:45.926474 16918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:45:45.926621 16918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:45:45.926718 16918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:45:45.962422 16918 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.65:41857
I20260929 16:45:45.962509 17065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.65:41857 every 8 connection(s)
I20260929 16:45:45.964257 16918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
I20260929 16:45:45.966601 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 16918
I20260929 16:45:45.967368 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.66:43983
--local_ip_for_outbound_sockets=127.0.197.66
--tserver_master_addrs=127.0.197.126:37141
--webserver_port=40259
--webserver_interface=127.0.197.66
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab
--principal=kudu/127.0.197.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:33025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:45:45 dist-test-slave-l31f krb5kdc[16296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.65@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:45:45.993500 17068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:45.967819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:53877 (local address 127.0.197.126:37141)
0929 16:45:45.968537 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0929 16:45:45.968558 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 16:45:45.969371 (+ 813us) server_negotiation.cc:407] Connection header received
0929 16:45:45.972765 (+ 3394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:45:45.972780 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:45:45.973176 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:45:45.973498 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:45:45.975740 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:45.977121 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:45.978876 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:45.980064 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:45.980302 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:45.984471 (+ 4169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:45:45.984481 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:45:45.984498 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:45:45.984554 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:45:45.987880 (+ 3326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:45:45.989167 (+ 1287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:45:45.989182 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:45:45.989188 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:45:45.989373 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:45:45.989968 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:45:45.989972 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:45:45.989974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:45:45.990583 (+ 609us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:45:45.990823 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0929 16:45:45.991717 (+ 894us) server_negotiation.cc:299] Negotiation successful
0929 16:45:45.992118 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":457,"threads_started":1}
I20260929 16:45:45.993500 17069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:45.968145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.65:53877)
0929 16:45:45.969123 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:45.969181 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 16:45:45.972065 (+ 2884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:45:45.973743 (+ 1678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:45:45.973757 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:45:45.974606 (+ 849us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:45:45.975456 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:45:45.975487 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:45:45.977371 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:45:45.977375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:45:45.978695 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:45:45.978710 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:45:45.980350 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:45:45.980356 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:45:45.980572 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:45.981343 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:45:45.981368 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:45:45.984097 (+ 2729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:45:45.988101 (+ 4004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:45:45.988115 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:45:45.988138 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:45:45.988941 (+ 803us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:45:45.989510 (+ 569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:45:45.989518 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:45:45.989524 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:45:45.989819 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:45:45.990758 (+ 939us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:45:45.990768 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:45:45.991214 (+ 446us) client_negotiation.cc:769] Sending connection context
0929 16:45:45.991502 (+ 288us) client_negotiation.cc:240] Negotiation successful
0929 16:45:45.992118 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":387,"threads_started":1}
I20260929 16:45:45.997262 17066 heartbeater.cc:347] Connected to a master server at 127.0.197.126:37141
I20260929 16:45:45.997610 17066 heartbeater.cc:464] Registering TS with master...
I20260929 16:45:45.998415 17066 heartbeater.cc:511] Master 127.0.197.126:37141 requested a full tablet report, sending...
I20260929 16:45:46.001127 16881 ts_manager.cc:194] Registered new tserver with Master: a1e2ec51ef5d43cbb18df0696c0d159d (127.0.197.65:41857)
I20260929 16:45:46.003358 16881 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.65@KRBTEST.COM'} at 127.0.197.65:53877
W20260929 16:45:46.328231 17070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:45:46.328557 17070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:45:46.328658 17070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:45:46.336596 17070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:45:46.336745 17070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:45:46.336869 17070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.66
I20260929 16:45:46.347455 17070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:33025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab
--principal=kudu/127.0.197.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.66:43983
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.197.66
--webserver_port=40259
--tserver_master_addrs=127.0.197.126:37141
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.66
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:45:46.348760 17070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:45:46.350101 17070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:45:46.364905 17075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:45:46.365096 17078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:45:46.364910 17076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:45:46.365973 17070 server_base.cc:1034] running on GCE node
I20260929 16:45:46.366669 17070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:45:46.368044 17070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:45:46.369349 17070 hybrid_clock.cc:648] HybridClock initialized: now 1790700346369291 us; error 49 us; skew 500 ppm
I20260929 16:45:46.369801 17070 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:45:46 dist-test-slave-l31f krb5kdc[16296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700346, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:45:46.374248 17070 init.cc:377] Logged in from keytab as kudu/127.0.197.66@KRBTEST.COM (short username kudu)
I20260929 16:45:46.376047 17070 webserver.cc:460] Webserver started at http://127.0.197.66:40259/ using document root <none> and password file <none>
I20260929 16:45:46.376837 17070 fs_manager.cc:359] Metadata directory not provided
I20260929 16:45:46.377015 17070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:45:46.383208 17070 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:45:46.386821 17085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:45:46.388543 17070 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:45:46.388789 17070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-1/wal
uuid: "ddc41210f2dc4a3189be59eea46a2e31"
format_stamp: "Formatted at 2026-09-29 16:45:23 on dist-test-slave-l31f"
server_key: "0ecface9c95a52c3d3d9898c4d68dc2c"
server_key_iv: "8173df979e7f0113ead6a4a34c678444"
server_key_version: "encryptionkey@0"
I20260929 16:45:46.389887 17070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:45:46.411759 17070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:45:46.416996 17070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:45:46.417315 17070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:45:46.418853 17070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:45:46.420765 17070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:45:46.420892 17070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:45:46.421018 17070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:45:46.421108 17070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:45:46.460829 17070 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.66:43983
I20260929 16:45:46.460893 17197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.66:43983 every 8 connection(s)
I20260929 16:45:46.462419 17070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
I20260929 16:45:46.467594 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 17070
I20260929 16:45:46.468344 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.67:39065
--local_ip_for_outbound_sockets=127.0.197.67
--tserver_master_addrs=127.0.197.126:37141
--webserver_port=39013
--webserver_interface=127.0.197.67
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab
--principal=kudu/127.0.197.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:33025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:45:46 dist-test-slave-l31f krb5kdc[16296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700346, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.66@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:45:46.488551 17068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:46.465717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:38493 (local address 127.0.197.126:37141)
0929 16:45:46.465944 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 16:45:46.465949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:45:46.467550 (+ 1601us) server_negotiation.cc:407] Connection header received
0929 16:45:46.469882 (+ 2332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:45:46.469889 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:45:46.470045 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:45:46.470179 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:45:46.472224 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:46.473093 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:46.475283 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:46.475759 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:46.475886 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:46.481351 (+ 5465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:45:46.481357 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:45:46.481361 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:45:46.481405 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:45:46.484402 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:45:46.485691 (+ 1289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:45:46.485696 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:45:46.485699 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:45:46.485845 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:45:46.486445 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:45:46.486449 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:45:46.486451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:45:46.486821 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:45:46.486939 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 16:45:46.488072 (+ 1133us) server_negotiation.cc:299] Negotiation successful
0929 16:45:46.488287 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 16:45:46.490527 17200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:46.466352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.66:38493)
0929 16:45:46.467300 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:46.467387 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0929 16:45:46.469505 (+ 2118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:45:46.470495 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:45:46.470505 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:45:46.471263 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:45:46.471988 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:45:46.472010 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:45:46.473304 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:45:46.473308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:45:46.475075 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:45:46.475098 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:45:46.476064 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:45:46.476077 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:45:46.476480 (+ 403us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:46.477308 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:45:46.477354 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:45:46.480974 (+ 3620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:45:46.484640 (+ 3666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:45:46.484656 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:45:46.484684 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:45:46.485525 (+ 841us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:45:46.485976 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:45:46.485985 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:45:46.485992 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:45:46.486304 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:45:46.486972 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:45:46.486984 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:45:46.487589 (+ 605us) client_negotiation.cc:769] Sending connection context
0929 16:45:46.487966 (+ 377us) client_negotiation.cc:240] Negotiation successful
0929 16:45:46.488714 (+ 748us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":340,"threads_started":1}
I20260929 16:45:46.492272 17198 heartbeater.cc:347] Connected to a master server at 127.0.197.126:37141
I20260929 16:45:46.492695 17198 heartbeater.cc:464] Registering TS with master...
I20260929 16:45:46.493883 17198 heartbeater.cc:511] Master 127.0.197.126:37141 requested a full tablet report, sending...
I20260929 16:45:46.495888 16881 ts_manager.cc:194] Registered new tserver with Master: ddc41210f2dc4a3189be59eea46a2e31 (127.0.197.66:43983)
I20260929 16:45:46.496749 16881 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.66@KRBTEST.COM'} at 127.0.197.66:38493
W20260929 16:45:46.826535 17201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:45:46.826979 17201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:45:46.827204 17201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:45:46.835083 17201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:45:46.835291 17201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:45:46.835474 17201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.67
I20260929 16:45:46.846071 17201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:33025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab
--principal=kudu/127.0.197.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.67:39065
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.197.67
--webserver_port=39013
--tserver_master_addrs=127.0.197.126:37141
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.67
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:45:46.847558 17201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:45:46.848971 17201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:45:46.867319 17206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:45:46.867686 17207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:45:46.868749 17209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:45:46.869215 17201 server_base.cc:1034] running on GCE node
I20260929 16:45:46.870146 17201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:45:46.871820 17201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:45:46.873037 17201 hybrid_clock.cc:648] HybridClock initialized: now 1790700346872967 us; error 51 us; skew 500 ppm
I20260929 16:45:46.873546 17201 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:45:46 dist-test-slave-l31f krb5kdc[16296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700346, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:45:46.877705 17201 init.cc:377] Logged in from keytab as kudu/127.0.197.67@KRBTEST.COM (short username kudu)
I20260929 16:45:46.880405 17201 webserver.cc:460] Webserver started at http://127.0.197.67:39013/ using document root <none> and password file <none>
I20260929 16:45:46.881767 17201 fs_manager.cc:359] Metadata directory not provided
I20260929 16:45:46.881980 17201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:45:46.891808 17201 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 16:45:46.896283 17216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:45:46.898427 17201 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:45:46.898659 17201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-2/wal
uuid: "ba9242ba774841c0b786fec741f15825"
format_stamp: "Formatted at 2026-09-29 16:45:23 on dist-test-slave-l31f"
server_key: "e6f6b765e1513e73a86fc5be594d7aaf"
server_key_iv: "878dbc3927fc939010fddd3e7521120a"
server_key_version: "encryptionkey@0"
I20260929 16:45:46.900144 17201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:45:46.925222 17201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:45:46.930254 17201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:45:46.930572 17201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:45:46.931751 17201 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:45:46.933211 17201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:45:46.933346 17201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:45:46.933480 17201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:45:46.933580 17201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:45:46.972379 17201 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.67:39065
I20260929 16:45:46.972438 17328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.67:39065 every 8 connection(s)
I20260929 16:45:46.973896 17201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
I20260929 16:45:46.978824 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 17201
Sep 29 16:45:46 dist-test-slave-l31f krb5kdc[16296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700346, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.67@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:45:47.001374 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:46.979948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56824 (local address 127.0.197.126:37141)
0929 16:45:46.980761 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0929 16:45:46.980767 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:45:46.980782 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:45:46.980819 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:45:46.980824 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:45:46.980866 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:45:46.980970 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:45:46.983161 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:46.984478 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:46.986631 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:46.987306 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:46.987479 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:46.991741 (+ 4262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:45:46.991749 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:45:46.991755 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:45:46.991810 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:45:46.997142 (+ 5332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:45:46.998416 (+ 1274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:45:46.998426 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:45:46.998432 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:45:46.998642 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:45:46.999488 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:45:46.999494 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:45:46.999500 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:45:46.999888 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:45:47.000161 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0929 16:45:47.000574 (+ 413us) server_negotiation.cc:299] Negotiation successful
0929 16:45:47.000877 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":81,"server-negotiator.queue_time_us":668,"spinlock_wait_cycles":37376,"thread_start_us":524,"threads_started":1}
I20260929 16:45:47.003569 17068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:46.976667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:58955 (local address 127.0.197.126:37141)
0929 16:45:46.976896 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 16:45:46.976901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:45:46.978345 (+ 1444us) server_negotiation.cc:407] Connection header received
0929 16:45:46.980761 (+ 2416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:45:46.980767 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:45:46.980853 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:45:46.980970 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:45:46.983386 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:46.984481 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:46.986990 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:46.987657 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:46.987769 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:46.992800 (+ 5031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:45:46.992805 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:45:46.992808 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:45:46.992843 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:45:46.997862 (+ 5019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:45:46.999160 (+ 1298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:45:46.999169 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:45:46.999174 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:45:46.999336 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:45:47.000925 (+ 1589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:45:47.000930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:45:47.000933 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:45:47.001558 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:45:47.001717 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 16:45:47.002898 (+ 1181us) server_negotiation.cc:299] Negotiation successful
0929 16:45:47.003230 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":89,"spinlock_wait_cycles":10752}
I20260929 16:45:47.005089 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:46.977241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.67:58955)
0929 16:45:46.978113 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:46.978155 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 16:45:46.980379 (+ 2224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:45:46.981342 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:45:46.981357 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:45:46.982212 (+ 855us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:45:46.983199 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:45:46.983223 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:45:46.984712 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:45:46.984718 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:45:46.986796 (+ 2078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:45:46.986812 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:45:46.987927 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:45:46.987936 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:45:46.988256 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:46.988943 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:45:46.988976 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:45:46.992537 (+ 3561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:45:46.998106 (+ 5569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:45:46.998123 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:45:46.998150 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:45:46.998965 (+ 815us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:45:46.999485 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:45:46.999490 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:45:46.999494 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:45:46.999815 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:45:47.001812 (+ 1997us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:45:47.001826 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:45:47.002395 (+ 569us) client_negotiation.cc:769] Sending connection context
0929 16:45:47.002784 (+ 389us) client_negotiation.cc:240] Negotiation successful
0929 16:45:47.003511 (+ 727us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":350,"threads_started":1}
I20260929 16:45:47.007233 17066 heartbeater.cc:503] Master 127.0.197.126:37141 was elected leader, sending a full tablet report...
I20260929 16:45:47.007471 17329 heartbeater.cc:347] Connected to a master server at 127.0.197.126:37141
I20260929 16:45:47.007861 17329 heartbeater.cc:464] Registering TS with master...
I20260929 16:45:47.008708 17329 heartbeater.cc:511] Master 127.0.197.126:37141 requested a full tablet report, sending...
I20260929 16:45:47.010322 16880 ts_manager.cc:194] Registered new tserver with Master: ba9242ba774841c0b786fec741f15825 (127.0.197.67:39065)
I20260929 16:45:47.011360 16880 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.67@KRBTEST.COM'} at 127.0.197.67:58955
I20260929 16:45:47.020366 789 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:45:47.028852 17068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:47.021439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56836 (local address 127.0.197.126:37141)
0929 16:45:47.021736 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0929 16:45:47.021745 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:45:47.022235 (+ 490us) server_negotiation.cc:407] Connection header received
0929 16:45:47.022591 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:45:47.022597 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:45:47.022879 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:45:47.023245 (+ 366us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:45:47.023802 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:47.024640 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:47.026416 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:47.027212 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:47.027350 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:47.028044 (+ 694us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:45:47.028299 (+ 255us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0929 16:45:47.028641 (+ 342us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56836: token expired
Metrics: {"server-negotiator.queue_time_us":113}
W20260929 16:45:47.029090 17068 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56836: token expired
W20260929 16:45:47.030139 16805 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.197.126:37141)
I20260929 16:45:47.045174 17068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:47.030718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56840 (local address 127.0.197.126:37141)
0929 16:45:47.030930 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 16:45:47.030937 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:45:47.031048 (+ 111us) server_negotiation.cc:407] Connection header received
0929 16:45:47.031258 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:45:47.031263 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:45:47.031336 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:45:47.031464 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:45:47.033037 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:47.033844 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:47.035455 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:47.035887 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:47.036031 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:47.039131 (+ 3100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:45:47.039137 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:45:47.039140 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:45:47.039183 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:45:47.042001 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:45:47.042892 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:45:47.042898 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:45:47.042902 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:45:47.043138 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:45:47.043535 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:45:47.043541 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:45:47.043545 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:45:47.043930 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:45:47.044220 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0929 16:45:47.044557 (+ 337us) server_negotiation.cc:299] Negotiation successful
0929 16:45:47.044821 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 16:45:47.049045 16880 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56840:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:45:47.056795 16880 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:45:47.085011 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:47.071315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60490 (local address 127.0.197.67:39065)
0929 16:45:47.073182 (+ 1867us) server_negotiation.cc:206] Beginning negotiation
0929 16:45:47.073197 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:45:47.075392 (+ 2195us) server_negotiation.cc:407] Connection header received
0929 16:45:47.075600 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:45:47.075609 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:45:47.075923 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:45:47.076152 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:45:47.077205 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:47.078967 (+ 1762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:47.082244 (+ 3277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:47.083763 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:47.083937 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:47.084179 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0929 16:45:47.084485 (+ 306us) server_negotiation.cc:299] Negotiation successful
0929 16:45:47.084648 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1638,"thread_start_us":1418,"threads_started":1}
I20260929 16:45:47.084987 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:47.074661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38770 (local address 127.0.197.66:43983)
0929 16:45:47.076041 (+ 1380us) server_negotiation.cc:206] Beginning negotiation
0929 16:45:47.076052 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:45:47.076730 (+ 678us) server_negotiation.cc:407] Connection header received
0929 16:45:47.076828 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:45:47.076837 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:45:47.077076 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:45:47.077860 (+ 784us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:45:47.078087 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:47.079539 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:47.081783 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:47.083320 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:47.083468 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:47.083751 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0929 16:45:47.084442 (+ 691us) server_negotiation.cc:299] Negotiation successful
0929 16:45:47.084632 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1209,"thread_start_us":1073,"threads_started":1}
I20260929 16:45:47.085060 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:47.070235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:41857 (local address 127.0.0.1:56506)
0929 16:45:47.072059 (+ 1824us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:47.072112 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 16:45:47.072340 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:45:47.077474 (+ 5134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:45:47.077482 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:45:47.077516 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:45:47.077967 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:45:47.077979 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:45:47.079967 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:45:47.079973 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:45:47.082223 (+ 2250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:45:47.082235 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:45:47.084221 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:45:47.084227 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:45:47.084407 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:47.084433 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 16:45:47.084488 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 16:45:47.084599 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1478,"mutex_wait_us":2,"thread_start_us":4686,"threads_started":1}
I20260929 16:45:47.085575 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:47.070504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56506 (local address 127.0.197.65:41857)
0929 16:45:47.076895 (+ 6391us) server_negotiation.cc:206] Beginning negotiation
0929 16:45:47.076901 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:45:47.076924 (+ 23us) server_negotiation.cc:407] Connection header received
0929 16:45:47.077001 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:45:47.077007 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:45:47.077234 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:45:47.078164 (+ 930us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:45:47.078403 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:47.079721 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:47.082390 (+ 2669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:47.083945 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:47.084730 (+ 785us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:47.084951 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 16:45:47.085026 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 16:45:47.085229 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6243,"thread_start_us":6087,"threads_started":1}
I20260929 16:45:47.086053 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:47.070477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.67:39065 (local address 127.0.0.1:60490)
0929 16:45:47.075250 (+ 4773us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:47.075266 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:45:47.075426 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:45:47.076513 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:45:47.076524 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:45:47.076560 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:45:47.076973 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:45:47.076985 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:45:47.079141 (+ 2156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:45:47.079146 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:45:47.082035 (+ 2889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:45:47.082051 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:45:47.083986 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:45:47.083993 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:45:47.084261 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:47.084285 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 16:45:47.085548 (+ 1263us) client_negotiation.cc:240] Negotiation successful
0929 16:45:47.085708 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4628,"mutex_wait_us":53,"thread_start_us":1167,"threads_started":1}
I20260929 16:45:47.086179 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:47.074051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.66:43983 (local address 127.0.0.1:38770)
0929 16:45:47.076467 (+ 2416us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:47.076484 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:45:47.076598 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:45:47.077236 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:45:47.077241 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:45:47.077265 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:45:47.077677 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:45:47.077693 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:45:47.079697 (+ 2004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:45:47.079702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:45:47.081597 (+ 1895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:45:47.081612 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:45:47.083798 (+ 2186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:45:47.083805 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:45:47.084140 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:47.084221 (+ 81us) client_negotiation.cc:769] Sending connection context
0929 16:45:47.084368 (+ 147us) client_negotiation.cc:240] Negotiation successful
0929 16:45:47.084585 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2290,"mutex_wait_us":139,"thread_start_us":2140,"threads_started":1}
I20260929 16:45:47.089694 17133 tablet_service.cc:1467] Processing CreateTablet for tablet abd2bd7043154599b9ad6148bef52fe4 (DEFAULT_TABLE table=test-table [id=b58ff4a715334a97953b81f95ba9ea61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:45:47.089717 17264 tablet_service.cc:1467] Processing CreateTablet for tablet abd2bd7043154599b9ad6148bef52fe4 (DEFAULT_TABLE table=test-table [id=b58ff4a715334a97953b81f95ba9ea61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:45:47.089717 17001 tablet_service.cc:1467] Processing CreateTablet for tablet abd2bd7043154599b9ad6148bef52fe4 (DEFAULT_TABLE table=test-table [id=b58ff4a715334a97953b81f95ba9ea61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:45:47.091114 17133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abd2bd7043154599b9ad6148bef52fe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:45:47.091141 17264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abd2bd7043154599b9ad6148bef52fe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:45:47.091156 17001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abd2bd7043154599b9ad6148bef52fe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:45:47.105351 17345 tablet_bootstrap.cc:492] T abd2bd7043154599b9ad6148bef52fe4 P ddc41210f2dc4a3189be59eea46a2e31: Bootstrap starting.
I20260929 16:45:47.109227 17345 tablet_bootstrap.cc:654] T abd2bd7043154599b9ad6148bef52fe4 P ddc41210f2dc4a3189be59eea46a2e31: Neither blocks nor log segments found. Creating new log.
I20260929 16:45:47.110476 17345 log.cc:824] T abd2bd7043154599b9ad6148bef52fe4 P ddc41210f2dc4a3189be59eea46a2e31: Log is configured to *not* fsync() on all Append() calls
I20260929 16:45:47.113723 17346 tablet_bootstrap.cc:492] T abd2bd7043154599b9ad6148bef52fe4 P a1e2ec51ef5d43cbb18df0696c0d159d: Bootstrap starting.
I20260929 16:45:47.115075 17347 tablet_bootstrap.cc:492] T abd2bd7043154599b9ad6148bef52fe4 P ba9242ba774841c0b786fec741f15825: Bootstrap starting.
I20260929 16:45:47.117892 17346 tablet_bootstrap.cc:654] T abd2bd7043154599b9ad6148bef52fe4 P a1e2ec51ef5d43cbb18df0696c0d159d: Neither blocks nor log segments found. Creating new log.
I20260929 16:45:47.119144 17346 log.cc:824] T abd2bd7043154599b9ad6148bef52fe4 P a1e2ec51ef5d43cbb18df0696c0d159d: Log is configured to *not* fsync() on all Append() calls
I20260929 16:45:47.119354 17347 tablet_bootstrap.cc:654] T abd2bd7043154599b9ad6148bef52fe4 P ba9242ba774841c0b786fec741f15825: Neither blocks nor log segments found. Creating new log.
I20260929 16:45:47.120466 17345 tablet_bootstrap.cc:492] T abd2bd7043154599b9ad6148bef52fe4 P ddc41210f2dc4a3189be59eea46a2e31: No bootstrap required, opened a new log
I20260929 16:45:47.120687 17347 log.cc:824] T abd2bd7043154599b9ad6148bef52fe4 P ba9242ba774841c0b786fec741f15825: Log is configured to *not* fsync() on all Append() calls
I20260929 16:45:47.120819 17345 ts_tablet_manager.cc:1397] T abd2bd7043154599b9ad6148bef52fe4 P ddc41210f2dc4a3189be59eea46a2e31: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.008s
I20260929 16:45:47.122519 17346 tablet_bootstrap.cc:492] T abd2bd7043154599b9ad6148bef52fe4 P a1e2ec51ef5d43cbb18df0696c0d159d: No bootstrap required, opened a new log
I20260929 16:45:47.122836 17346 ts_tablet_manager.cc:1397] T abd2bd7043154599b9ad6148bef52fe4 P a1e2ec51ef5d43cbb18df0696c0d159d: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260929 16:45:47.124433 17347 tablet_bootstrap.cc:492] T abd2bd7043154599b9ad6148bef52fe4 P ba9242ba774841c0b786fec741f15825: No bootstrap required, opened a new log
I20260929 16:45:47.124760 17347 ts_tablet_manager.cc:1397] T abd2bd7043154599b9ad6148bef52fe4 P ba9242ba774841c0b786fec741f15825: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260929 16:45:47.129045 17345 raft_consensus.cc:348] T abd2bd7043154599b9ad6148bef52fe4 P ddc41210f2dc4a3189be59eea46a2e31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e2ec51ef5d43cbb18df0696c0d159d" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 41857 } } peers { permanent_uuid: "ba9242ba774841c0b786fec741f15825" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 39065 } } peers { permanent_uuid: "ddc41210f2dc4a3189be59eea46a2e31" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43983 } }
I20260929 16:45:47.129531 17345 raft_consensus.cc:374] T abd2bd7043154599b9ad6148bef52fe4 P ddc41210f2dc4a3189be59eea46a2e31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:45:47.129652 17345 raft_consensus.cc:729] T abd2bd7043154599b9ad6148bef52fe4 P ddc41210f2dc4a3189be59eea46a2e31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddc41210f2dc4a3189be59eea46a2e31, State: Initialized, Role: FOLLOWER
I20260929 16:45:47.130344 17345 consensus_queue.cc:260] T abd2bd7043154599b9ad6148bef52fe4 P ddc41210f2dc4a3189be59eea46a2e31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e2ec51ef5d43cbb18df0696c0d159d" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 41857 } } peers { permanent_uuid: "ba9242ba774841c0b786fec741f15825" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 39065 } } peers { permanent_uuid: "ddc41210f2dc4a3189be59eea46a2e31" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43983 } }
I20260929 16:45:47.131629 17198 heartbeater.cc:503] Master 127.0.197.126:37141 was elected leader, sending a full tablet report...
I20260929 16:45:47.132391 17345 ts_tablet_manager.cc:1428] T abd2bd7043154599b9ad6148bef52fe4 P ddc41210f2dc4a3189be59eea46a2e31: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260929 16:45:47.134469 17346 raft_consensus.cc:348] T abd2bd7043154599b9ad6148bef52fe4 P a1e2ec51ef5d43cbb18df0696c0d159d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e2ec51ef5d43cbb18df0696c0d159d" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 41857 } } peers { permanent_uuid: "ba9242ba774841c0b786fec741f15825" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 39065 } } peers { permanent_uuid: "ddc41210f2dc4a3189be59eea46a2e31" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43983 } }
I20260929 16:45:47.135037 17346 raft_consensus.cc:374] T abd2bd7043154599b9ad6148bef52fe4 P a1e2ec51ef5d43cbb18df0696c0d159d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:45:47.135150 17346 raft_consensus.cc:729] T abd2bd7043154599b9ad6148bef52fe4 P a1e2ec51ef5d43cbb18df0696c0d159d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1e2ec51ef5d43cbb18df0696c0d159d, State: Initialized, Role: FOLLOWER
I20260929 16:45:47.135946 17346 consensus_queue.cc:260] T abd2bd7043154599b9ad6148bef52fe4 P a1e2ec51ef5d43cbb18df0696c0d159d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e2ec51ef5d43cbb18df0696c0d159d" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 41857 } } peers { permanent_uuid: "ba9242ba774841c0b786fec741f15825" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 39065 } } peers { permanent_uuid: "ddc41210f2dc4a3189be59eea46a2e31" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43983 } }
I20260929 16:45:47.136250 17347 raft_consensus.cc:348] T abd2bd7043154599b9ad6148bef52fe4 P ba9242ba774841c0b786fec741f15825 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e2ec51ef5d43cbb18df0696c0d159d" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 41857 } } peers { permanent_uuid: "ba9242ba774841c0b786fec741f15825" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 39065 } } peers { permanent_uuid: "ddc41210f2dc4a3189be59eea46a2e31" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43983 } }
I20260929 16:45:47.136780 17347 raft_consensus.cc:374] T abd2bd7043154599b9ad6148bef52fe4 P ba9242ba774841c0b786fec741f15825 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:45:47.136897 17347 raft_consensus.cc:729] T abd2bd7043154599b9ad6148bef52fe4 P ba9242ba774841c0b786fec741f15825 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba9242ba774841c0b786fec741f15825, State: Initialized, Role: FOLLOWER
I20260929 16:45:47.137554 17347 consensus_queue.cc:260] T abd2bd7043154599b9ad6148bef52fe4 P ba9242ba774841c0b786fec741f15825 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e2ec51ef5d43cbb18df0696c0d159d" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 41857 } } peers { permanent_uuid: "ba9242ba774841c0b786fec741f15825" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 39065 } } peers { permanent_uuid: "ddc41210f2dc4a3189be59eea46a2e31" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43983 } }
I20260929 16:45:47.138057 17346 ts_tablet_manager.cc:1428] T abd2bd7043154599b9ad6148bef52fe4 P a1e2ec51ef5d43cbb18df0696c0d159d: Time spent starting tablet: real 0.015s user 0.004s sys 0.012s
I20260929 16:45:47.139889 17329 heartbeater.cc:503] Master 127.0.197.126:37141 was elected leader, sending a full tablet report...
I20260929 16:45:47.140659 17347 ts_tablet_manager.cc:1428] T abd2bd7043154599b9ad6148bef52fe4 P ba9242ba774841c0b786fec741f15825: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
W20260929 16:45:47.215041 17199 tablet.cc:2369] T abd2bd7043154599b9ad6148bef52fe4 P ddc41210f2dc4a3189be59eea46a2e31: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:45:47.215888 17353 raft_consensus.cc:482] T abd2bd7043154599b9ad6148bef52fe4 P ba9242ba774841c0b786fec741f15825 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:45:47.216234 17353 raft_consensus.cc:504] T abd2bd7043154599b9ad6148bef52fe4 P ba9242ba774841c0b786fec741f15825 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e2ec51ef5d43cbb18df0696c0d159d" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 41857 } } peers { permanent_uuid: "ba9242ba774841c0b786fec741f15825" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 39065 } } peers { permanent_uuid: "ddc41210f2dc4a3189be59eea46a2e31" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43983 } }
W20260929 16:45:47.216801 17067 tablet.cc:2369] T abd2bd7043154599b9ad6148bef52fe4 P a1e2ec51ef5d43cbb18df0696c0d159d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:45:47.218142 17353 leader_election.cc:288] T abd2bd7043154599b9ad6148bef52fe4 P ba9242ba774841c0b786fec741f15825 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1e2ec51ef5d43cbb18df0696c0d159d (127.0.197.65:41857), ddc41210f2dc4a3189be59eea46a2e31 (127.0.197.66:43983)
I20260929 16:45:47.225262 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:47.218740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:41599 (local address 127.0.197.65:41857)
0929 16:45:47.218978 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 16:45:47.218984 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:45:47.219056 (+ 72us) server_negotiation.cc:407] Connection header received
0929 16:45:47.219280 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:45:47.219287 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:45:47.219380 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:45:47.219512 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:45:47.219864 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:47.221029 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:47.222995 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:47.224339 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:47.224475 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:47.224539 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 16:45:47.224930 (+ 391us) server_negotiation.cc:299] Negotiation successful
0929 16:45:47.225040 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260929 16:45:47.225350 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:47.218486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:41857 (local address 127.0.197.67:41599)
0929 16:45:47.218827 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:47.218851 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 16:45:47.219075 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:45:47.219544 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:45:47.219548 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:45:47.219572 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:45:47.219741 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:45:47.219748 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:45:47.221183 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:45:47.221189 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:45:47.222798 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:45:47.222812 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:45:47.224531 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:45:47.224538 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:45:47.224743 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:47.224780 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 16:45:47.224891 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 16:45:47.225049 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171}
W20260929 16:45:47.225672 17330 tablet.cc:2369] T abd2bd7043154599b9ad6148bef52fe4 P ba9242ba774841c0b786fec741f15825: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:45:47.226646 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:47.218758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:35935 (local address 127.0.197.66:43983)
0929 16:45:47.218978 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 16:45:47.218985 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:45:47.219742 (+ 757us) server_negotiation.cc:407] Connection header received
0929 16:45:47.219974 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:45:47.219978 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:45:47.220039 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:45:47.220109 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:45:47.220802 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:47.221805 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:47.224175 (+ 2370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:47.225382 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:47.225554 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:47.225634 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 16:45:47.226241 (+ 607us) server_negotiation.cc:299] Negotiation successful
0929 16:45:47.226381 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 16:45:47.226691 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:47.218601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.66:43983 (local address 127.0.197.67:35935)
0929 16:45:47.219616 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:47.219636 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:45:47.219798 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:45:47.220164 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:45:47.220169 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:45:47.220199 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:45:47.220592 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:45:47.220605 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:45:47.221951 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:45:47.221955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:45:47.223965 (+ 2010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:45:47.223981 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:45:47.225810 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:45:47.225815 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:45:47.226030 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:47.226071 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 16:45:47.226197 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 16:45:47.226335 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":851,"thread_start_us":640,"threads_started":1}
I20260929 16:45:47.226691 17021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd2bd7043154599b9ad6148bef52fe4" candidate_uuid: "ba9242ba774841c0b786fec741f15825" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1e2ec51ef5d43cbb18df0696c0d159d" is_pre_election: true
I20260929 16:45:47.227347 17021 raft_consensus.cc:2454] T abd2bd7043154599b9ad6148bef52fe4 P a1e2ec51ef5d43cbb18df0696c0d159d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba9242ba774841c0b786fec741f15825 in term 0.
I20260929 16:45:47.227777 17153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd2bd7043154599b9ad6148bef52fe4" candidate_uuid: "ba9242ba774841c0b786fec741f15825" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddc41210f2dc4a3189be59eea46a2e31" is_pre_election: true
I20260929 16:45:47.228111 17220 leader_election.cc:302] T abd2bd7043154599b9ad6148bef52fe4 P ba9242ba774841c0b786fec741f15825 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1e2ec51ef5d43cbb18df0696c0d159d, ba9242ba774841c0b786fec741f15825; no voters:
I20260929 16:45:47.228312 17153 raft_consensus.cc:2454] T abd2bd7043154599b9ad6148bef52fe4 P ddc41210f2dc4a3189be59eea46a2e31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba9242ba774841c0b786fec741f15825 in term 0.
I20260929 16:45:47.228554 17353 raft_consensus.cc:2781] T abd2bd7043154599b9ad6148bef52fe4 P ba9242ba774841c0b786fec741f15825 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:45:47.228732 17353 raft_consensus.cc:482] T abd2bd7043154599b9ad6148bef52fe4 P ba9242ba774841c0b786fec741f15825 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:45:47.228865 17353 raft_consensus.cc:3037] T abd2bd7043154599b9ad6148bef52fe4 P ba9242ba774841c0b786fec741f15825 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:45:47.231647 17353 raft_consensus.cc:504] T abd2bd7043154599b9ad6148bef52fe4 P ba9242ba774841c0b786fec741f15825 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e2ec51ef5d43cbb18df0696c0d159d" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 41857 } } peers { permanent_uuid: "ba9242ba774841c0b786fec741f15825" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 39065 } } peers { permanent_uuid: "ddc41210f2dc4a3189be59eea46a2e31" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43983 } }
I20260929 16:45:47.232419 17353 leader_election.cc:288] T abd2bd7043154599b9ad6148bef52fe4 P ba9242ba774841c0b786fec741f15825 [CANDIDATE]: Term 1 election: Requested vote from peers a1e2ec51ef5d43cbb18df0696c0d159d (127.0.197.65:41857), ddc41210f2dc4a3189be59eea46a2e31 (127.0.197.66:43983)
I20260929 16:45:47.233273 17021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd2bd7043154599b9ad6148bef52fe4" candidate_uuid: "ba9242ba774841c0b786fec741f15825" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1e2ec51ef5d43cbb18df0696c0d159d"
I20260929 16:45:47.233399 17153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd2bd7043154599b9ad6148bef52fe4" candidate_uuid: "ba9242ba774841c0b786fec741f15825" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddc41210f2dc4a3189be59eea46a2e31"
I20260929 16:45:47.233513 17021 raft_consensus.cc:3037] T abd2bd7043154599b9ad6148bef52fe4 P a1e2ec51ef5d43cbb18df0696c0d159d [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:45:47.233693 17153 raft_consensus.cc:3037] T abd2bd7043154599b9ad6148bef52fe4 P ddc41210f2dc4a3189be59eea46a2e31 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:45:47.235975 17021 raft_consensus.cc:2454] T abd2bd7043154599b9ad6148bef52fe4 P a1e2ec51ef5d43cbb18df0696c0d159d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba9242ba774841c0b786fec741f15825 in term 1.
I20260929 16:45:47.236083 17153 raft_consensus.cc:2454] T abd2bd7043154599b9ad6148bef52fe4 P ddc41210f2dc4a3189be59eea46a2e31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba9242ba774841c0b786fec741f15825 in term 1.
I20260929 16:45:47.236625 17218 leader_election.cc:302] T abd2bd7043154599b9ad6148bef52fe4 P ba9242ba774841c0b786fec741f15825 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba9242ba774841c0b786fec741f15825, ddc41210f2dc4a3189be59eea46a2e31; no voters:
I20260929 16:45:47.237053 17353 raft_consensus.cc:2781] T abd2bd7043154599b9ad6148bef52fe4 P ba9242ba774841c0b786fec741f15825 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:45:47.237565 17353 raft_consensus.cc:686] T abd2bd7043154599b9ad6148bef52fe4 P ba9242ba774841c0b786fec741f15825 [term 1 LEADER]: Becoming Leader. State: Replica: ba9242ba774841c0b786fec741f15825, State: Running, Role: LEADER
I20260929 16:45:47.238094 17353 consensus_queue.cc:237] T abd2bd7043154599b9ad6148bef52fe4 P ba9242ba774841c0b786fec741f15825 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e2ec51ef5d43cbb18df0696c0d159d" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 41857 } } peers { permanent_uuid: "ba9242ba774841c0b786fec741f15825" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 39065 } } peers { permanent_uuid: "ddc41210f2dc4a3189be59eea46a2e31" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43983 } }
I20260929 16:45:47.243150 16881 catalog_manager.cc:5495] T abd2bd7043154599b9ad6148bef52fe4 P ba9242ba774841c0b786fec741f15825 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba9242ba774841c0b786fec741f15825 (127.0.197.67). New cstate: current_term: 1 leader_uuid: "ba9242ba774841c0b786fec741f15825" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e2ec51ef5d43cbb18df0696c0d159d" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 41857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba9242ba774841c0b786fec741f15825" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 39065 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddc41210f2dc4a3189be59eea46a2e31" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43983 } health_report { overall_health: UNKNOWN } } }
I20260929 16:45:47.281198 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:47.274755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60500 (local address 127.0.197.67:39065)
0929 16:45:47.274966 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 16:45:47.274971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:45:47.274990 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:45:47.275162 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:45:47.275166 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:45:47.275257 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:45:47.275385 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:45:47.275791 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:47.276916 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:47.278486 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:47.279288 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:47.279398 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:47.279967 (+ 569us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:45:47.280521 (+ 554us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:45:47.280652 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 16:45:47.280772 (+ 120us) server_negotiation.cc:299] Negotiation successful
0929 16:45:47.280850 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 16:45:47.290127 17021 raft_consensus.cc:1261] T abd2bd7043154599b9ad6148bef52fe4 P a1e2ec51ef5d43cbb18df0696c0d159d [term 1 FOLLOWER]: Refusing update from remote peer ba9242ba774841c0b786fec741f15825: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:45:47.290133 17153 raft_consensus.cc:1261] T abd2bd7043154599b9ad6148bef52fe4 P ddc41210f2dc4a3189be59eea46a2e31 [term 1 FOLLOWER]: Refusing update from remote peer ba9242ba774841c0b786fec741f15825: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:45:47.291316 17355 consensus_queue.cc:1035] T abd2bd7043154599b9ad6148bef52fe4 P ba9242ba774841c0b786fec741f15825 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddc41210f2dc4a3189be59eea46a2e31" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 43983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:45:47.291690 17353 consensus_queue.cc:1035] T abd2bd7043154599b9ad6148bef52fe4 P ba9242ba774841c0b786fec741f15825 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1e2ec51ef5d43cbb18df0696c0d159d" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 41857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:45:47.308622 17358 mvcc.cc:205] Tried to move back new op lower bound from 7334708622489726976 to 7334708622291734528. Current Snapshot: MvccSnapshot[applied={T|T < 7334708622489726976}]
I20260929 16:45:47.322664 16881 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:45:47.322996 16881 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:45:47.327072 16881 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e: Sending DeleteTablet for 3 replicas of tablet abd2bd7043154599b9ad6148bef52fe4
I20260929 16:45:47.328629 17001 tablet_service.cc:1514] Processing DeleteTablet for tablet abd2bd7043154599b9ad6148bef52fe4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:45:47 UTC) from {username='kudu', principal='kudu/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:56506
I20260929 16:45:47.328711 17264 tablet_service.cc:1514] Processing DeleteTablet for tablet abd2bd7043154599b9ad6148bef52fe4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:45:47 UTC) from {username='kudu', principal='kudu/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:60490
I20260929 16:45:47.328697 17133 tablet_service.cc:1514] Processing DeleteTablet for tablet abd2bd7043154599b9ad6148bef52fe4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:45:47 UTC) from {username='kudu', principal='kudu/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:38770
I20260929 16:45:47.329967 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 16918
I20260929 16:45:47.348884 17369 tablet_replica.cc:330] T abd2bd7043154599b9ad6148bef52fe4 P ddc41210f2dc4a3189be59eea46a2e31: stopping tablet replica
I20260929 16:45:47.349184 17368 tablet_replica.cc:330] T abd2bd7043154599b9ad6148bef52fe4 P a1e2ec51ef5d43cbb18df0696c0d159d: stopping tablet replica
I20260929 16:45:47.349649 17369 raft_consensus.cc:2229] T abd2bd7043154599b9ad6148bef52fe4 P ddc41210f2dc4a3189be59eea46a2e31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:45:47.349790 17368 raft_consensus.cc:2229] T abd2bd7043154599b9ad6148bef52fe4 P a1e2ec51ef5d43cbb18df0696c0d159d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:45:47.350335 17369 raft_consensus.cc:2258] T abd2bd7043154599b9ad6148bef52fe4 P ddc41210f2dc4a3189be59eea46a2e31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:45:47.350417 17368 raft_consensus.cc:2258] T abd2bd7043154599b9ad6148bef52fe4 P a1e2ec51ef5d43cbb18df0696c0d159d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:45:47.351333 17370 tablet_replica.cc:330] T abd2bd7043154599b9ad6148bef52fe4 P ba9242ba774841c0b786fec741f15825: stopping tablet replica
I20260929 16:45:47.351919 17370 raft_consensus.cc:2229] T abd2bd7043154599b9ad6148bef52fe4 P ba9242ba774841c0b786fec741f15825 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:45:47.352751 17370 raft_consensus.cc:2258] T abd2bd7043154599b9ad6148bef52fe4 P ba9242ba774841c0b786fec741f15825 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:45:47.352996 17368 ts_tablet_manager.cc:1905] T abd2bd7043154599b9ad6148bef52fe4 P a1e2ec51ef5d43cbb18df0696c0d159d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:45:47.355365 17369 ts_tablet_manager.cc:1905] T abd2bd7043154599b9ad6148bef52fe4 P ddc41210f2dc4a3189be59eea46a2e31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:45:47.355996 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:47.331618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56512 (local address 127.0.197.65:41857)
0929 16:45:47.331834 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 16:45:47.331844 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:45:47.331871 (+ 27us) server_negotiation.cc:407] Connection header received
0929 16:45:47.331954 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:45:47.331961 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:45:47.332100 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:45:47.332233 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:45:47.334411 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:47.336063 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:47.338118 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:47.339641 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:47.340071 (+ 430us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:47.345691 (+ 5620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:45:47.345722 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:45:47.345751 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:45:47.345830 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:45:47.351370 (+ 5540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:45:47.352556 (+ 1186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:45:47.352567 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:45:47.352571 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:45:47.352773 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:45:47.353544 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:45:47.353552 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:45:47.353555 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:45:47.354385 (+ 830us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:45:47.354741 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0929 16:45:47.355261 (+ 520us) server_negotiation.cc:299] Negotiation successful
0929 16:45:47.355538 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 16:45:47.356738 17061 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:45:47.357982 17370 ts_tablet_manager.cc:1905] T abd2bd7043154599b9ad6148bef52fe4 P ba9242ba774841c0b786fec741f15825: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:45:47.365365 17369 ts_tablet_manager.cc:1918] T abd2bd7043154599b9ad6148bef52fe4 P ddc41210f2dc4a3189be59eea46a2e31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:45:47.365631 17369 log.cc:1196] T abd2bd7043154599b9ad6148bef52fe4 P ddc41210f2dc4a3189be59eea46a2e31: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-1/wal/wals/abd2bd7043154599b9ad6148bef52fe4
I20260929 16:45:47.365926 17370 ts_tablet_manager.cc:1918] T abd2bd7043154599b9ad6148bef52fe4 P ba9242ba774841c0b786fec741f15825: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:45:47.366261 17370 log.cc:1196] T abd2bd7043154599b9ad6148bef52fe4 P ba9242ba774841c0b786fec741f15825: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-2/wal/wals/abd2bd7043154599b9ad6148bef52fe4
I20260929 16:45:47.366320 17369 ts_tablet_manager.cc:1939] T abd2bd7043154599b9ad6148bef52fe4 P ddc41210f2dc4a3189be59eea46a2e31: Deleting consensus metadata
I20260929 16:45:47.366886 17370 ts_tablet_manager.cc:1939] T abd2bd7043154599b9ad6148bef52fe4 P ba9242ba774841c0b786fec741f15825: Deleting consensus metadata
I20260929 16:45:47.368929 16866 catalog_manager.cc:4841] TS ba9242ba774841c0b786fec741f15825 (127.0.197.67:39065): tablet abd2bd7043154599b9ad6148bef52fe4 (table test-table [id=b58ff4a715334a97953b81f95ba9ea61]) successfully deleted
I20260929 16:45:47.369326 16866 catalog_manager.cc:4841] TS ddc41210f2dc4a3189be59eea46a2e31 (127.0.197.66:43983): tablet abd2bd7043154599b9ad6148bef52fe4 (table test-table [id=b58ff4a715334a97953b81f95ba9ea61]) successfully deleted
I20260929 16:45:47.491374 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 16918
W20260929 16:45:47.518868 16868 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.197.65:41857 (error 108)
I20260929 16:45:47.519256 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 17070
I20260929 16:45:47.521092 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:47.520562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:41857 (local address 127.0.0.1:56520)
0929 16:45:47.520763 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:47.520887 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.65:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260929 16:45:47.521380 16868 catalog_manager.cc:4584] TS a1e2ec51ef5d43cbb18df0696c0d159d (127.0.197.65:41857): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet abd2bd7043154599b9ad6148bef52fe4: Network error: Client connection negotiation failed: client connection to 127.0.197.65:41857: connect: Connection refused (error 111)
I20260929 16:45:47.542042 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:47.520099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38776 (local address 127.0.197.66:43983)
0929 16:45:47.520274 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 16:45:47.520279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:45:47.520409 (+ 130us) server_negotiation.cc:407] Connection header received
0929 16:45:47.520601 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:45:47.520608 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:45:47.520708 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:45:47.520821 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:45:47.522567 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:47.523924 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:47.525454 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:47.526095 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:47.526248 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:47.534018 (+ 7770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:45:47.534042 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:45:47.534065 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:45:47.534122 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:45:47.538364 (+ 4242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:45:47.539456 (+ 1092us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:45:47.539469 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:45:47.539479 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:45:47.539672 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:45:47.540152 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:45:47.540157 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:45:47.540160 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:45:47.540706 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:45:47.541027 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0929 16:45:47.541391 (+ 364us) server_negotiation.cc:299] Negotiation successful
0929 16:45:47.541662 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:45:47.543051 17193 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:45:47.575352 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:47.574729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:41857 (local address 127.0.0.1:56526)
0929 16:45:47.575028 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:47.575149 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.65:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260929 16:45:47.576678 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:47.576218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:41857 (local address 127.0.0.1:56532)
0929 16:45:47.576405 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:47.576509 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.65:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 16:45:47.637053 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 17070
I20260929 16:45:47.637447 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:47.636854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:41857 (local address 127.0.0.1:56536)
0929 16:45:47.637149 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:47.637263 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.65:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20260929 16:45:47.639690 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:47.638321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:41857 (local address 127.0.0.1:56540)
0929 16:45:47.639458 (+ 1137us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:47.639548 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.65:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1023}
I20260929 16:45:47.657132 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 17201
I20260929 16:45:47.675702 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:47.658041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60516 (local address 127.0.197.67:39065)
0929 16:45:47.658255 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 16:45:47.658263 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:45:47.658284 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:45:47.658531 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:45:47.658537 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:45:47.658629 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:45:47.658762 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:45:47.660270 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:47.661483 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:47.662905 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:45:47.663926 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:45:47.664092 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:45:47.667542 (+ 3450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:45:47.667567 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:45:47.667615 (+ 48us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:45:47.667690 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:45:47.672108 (+ 4418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:45:47.673111 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:45:47.673123 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:45:47.673127 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:45:47.673320 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:45:47.673812 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:45:47.673818 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:45:47.673822 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:45:47.674339 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:45:47.674701 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0929 16:45:47.674980 (+ 279us) server_negotiation.cc:299] Negotiation successful
0929 16:45:47.675284 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 16:45:47.676628 17324 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:45:47.720603 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:47.720028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:41857 (local address 127.0.0.1:56548)
0929 16:45:47.720298 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:47.720409 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.65:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260929 16:45:47.721942 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:47.721501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:41857 (local address 127.0.0.1:56552)
0929 16:45:47.721729 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:47.721819 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.65:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260929 16:45:47.807331 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 17201
I20260929 16:45:47.828172 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 16848
I20260929 16:45:47.829990 16911 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:45:47.948416 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:45:47.947841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:41857 (local address 127.0.0.1:56568)
0929 16:45:47.948030 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0929 16:45:47.948143 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.65:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 16:45:47.948381 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 16848
I20260929 16:46:07.972827 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:37141
--webserver_interface=127.0.197.126
--webserver_port=35897
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:33025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.126:37141
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:46:08.333880 17376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:46:08.334229 17376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:46:08.334326 17376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:46:08.341821 17376 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 16:46:08.341960 17376 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 16:46:08.342002 17376 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 16:46:08.342047 17376 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:46:08.342095 17376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:46:08.342168 17376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:46:08.342217 17376 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:46:08.342267 17376 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:46:08.353646 17376 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:33025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.197.126:37141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:37141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.197.126
--webserver_port=35897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:46:08.355110 17376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:46:08.356473 17376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:46:08.370599 17381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:08.370548 17382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:08.370476 17384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:46:08.370875 17376 server_base.cc:1034] running on GCE node
I20260929 16:46:08.371990 17376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:46:08.373615 17376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:46:08.374910 17376 hybrid_clock.cc:648] HybridClock initialized: now 1790700368374822 us; error 63 us; skew 500 ppm
I20260929 16:46:08.375332 17376 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:46:08 dist-test-slave-l31f krb5kdc[16296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700368, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:46:08.378894 17376 init.cc:377] Logged in from keytab as kudu/127.0.197.126@KRBTEST.COM (short username kudu)
I20260929 16:46:08.380594 17376 webserver.cc:460] Webserver started at http://127.0.197.126:35897/ using document root <none> and password file <none>
I20260929 16:46:08.381356 17376 fs_manager.cc:359] Metadata directory not provided
I20260929 16:46:08.381466 17376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:46:08.387749 17376 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 16:46:08.391083 17391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:08.392656 17376 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:46:08.392884 17376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "5014d3abe04b487cab15866468715f4e"
format_stamp: "Formatted at 2026-09-29 16:45:22 on dist-test-slave-l31f"
server_key: "37b457cf171f0a8a8f28d7261010da4e"
server_key_iv: "996c493fa1f8594a13099f8a85ac83a5"
server_key_version: "encryptionkey@0"
I20260929 16:46:08.393949 17376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:46:08.410800 17376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:46:08.415985 17376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:46:08.416358 17376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:46:08.436913 17376 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.126:37141
I20260929 16:46:08.436920 17442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.126:37141 every 8 connection(s)
I20260929 16:46:08.438578 17376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:46:08.439817 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 17376
I20260929 16:46:08.440729 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.65:41857
--local_ip_for_outbound_sockets=127.0.197.65
--tserver_master_addrs=127.0.197.126:37141
--webserver_port=43127
--webserver_interface=127.0.197.65
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab
--principal=kudu/127.0.197.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:33025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:46:08.446792 17443 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:46:08.450829 17443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e: Bootstrap starting.
I20260929 16:46:08.472744 17443 log.cc:824] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e: Log is configured to *not* fsync() on all Append() calls
I20260929 16:46:08.489661 17443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:46:08.490286 17443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e: Bootstrap complete.
I20260929 16:46:08.500325 17443 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5014d3abe04b487cab15866468715f4e" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 37141 } }
I20260929 16:46:08.501317 17443 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5014d3abe04b487cab15866468715f4e, State: Initialized, Role: FOLLOWER
I20260929 16:46:08.502014 17443 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5014d3abe04b487cab15866468715f4e" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 37141 } }
I20260929 16:46:08.502280 17443 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:46:08.502414 17443 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:46:08.502609 17443 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e [term 2 FOLLOWER]: Advancing to term 3
I20260929 16:46:08.507532 17443 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5014d3abe04b487cab15866468715f4e" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 37141 } }
I20260929 16:46:08.508216 17443 leader_election.cc:302] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5014d3abe04b487cab15866468715f4e; no voters:
I20260929 16:46:08.508962 17443 leader_election.cc:288] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 16:46:08.509116 17447 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e [term 3 FOLLOWER]: Leader election won for term 3
I20260929 16:46:08.509868 17447 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e [term 3 LEADER]: Becoming Leader. State: Replica: 5014d3abe04b487cab15866468715f4e, State: Running, Role: LEADER
I20260929 16:46:08.510517 17447 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5014d3abe04b487cab15866468715f4e" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 37141 } }
I20260929 16:46:08.511349 17443 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:46:08.513650 17448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5014d3abe04b487cab15866468715f4e. Latest consensus state: current_term: 3 leader_uuid: "5014d3abe04b487cab15866468715f4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5014d3abe04b487cab15866468715f4e" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 37141 } } }
I20260929 16:46:08.514380 17448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e [sys.catalog]: This master's current role is: LEADER
I20260929 16:46:08.513616 17449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "5014d3abe04b487cab15866468715f4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5014d3abe04b487cab15866468715f4e" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 37141 } } }
I20260929 16:46:08.514669 17449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e [sys.catalog]: This master's current role is: LEADER
I20260929 16:46:08.530598 17452 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:46:08.541110 17452 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:46:08.542744 17452 catalog_manager.cc:1259] Loaded cluster ID: d7396f1a6df84581bbb913b096f068a6
I20260929 16:46:08.542872 17452 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:46:08.547369 17452 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:46:08.550608 17452 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e: Loaded TSK: 0
I20260929 16:46:08.551034 17452 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5014d3abe04b487cab15866468715f4e: Loaded TSK: 1
F20260929 16:46:08.552325 17452 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f996b9eb9ad google::LogMessage::Fail() at ??:0
@ 0x7f996b9ee88a google::LogMessage::SendToLog() at ??:0
@ 0x7f996b9eb4de google::LogMessage::Flush() at ??:0
@ 0x7f996b9eca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f996f74f7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f996f73e73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f998141a1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f9981419783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f99814a52a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f99814a50c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f99814ef5e9 std::function<>::operator()() at ??:0
@ 0x7f998141dcdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f998141bc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f99814a3610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f99814a3439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f996e637345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f996e63a91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f996e63a799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f996e615eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f99718ef6db start_thread at ??:0
@ 0x7f996a88561f clone at ??:0
W20260929 16:46:08.884347 17445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:46:08.884688 17445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:46:08.884852 17445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:46:08.892920 17445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:46:08.893071 17445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:46:08.893277 17445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.65
I20260929 16:46:08.904920 17445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:33025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab
--principal=kudu/127.0.197.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.65:41857
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.197.65
--webserver_port=43127
--tserver_master_addrs=127.0.197.126:37141
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.65
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:46:08.906450 17445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:46:08.907981 17445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:46:08.923226 17493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:08.923297 17491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:08.923264 17490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:46:08.924067 17445 server_base.cc:1034] running on GCE node
I20260929 16:46:08.924785 17445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:46:08.925966 17445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:46:08.927248 17445 hybrid_clock.cc:648] HybridClock initialized: now 1790700368927199 us; error 42 us; skew 500 ppm
I20260929 16:46:08.927661 17445 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:46:08 dist-test-slave-l31f krb5kdc[16296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700368, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:46:08.931389 17445 init.cc:377] Logged in from keytab as kudu/127.0.197.65@KRBTEST.COM (short username kudu)
I20260929 16:46:08.933137 17445 webserver.cc:460] Webserver started at http://127.0.197.65:43127/ using document root <none> and password file <none>
I20260929 16:46:08.933940 17445 fs_manager.cc:359] Metadata directory not provided
I20260929 16:46:08.934114 17445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:46:08.940311 17445 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 16:46:08.943760 17500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:08.945385 17445 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:46:08.945642 17445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-0/wal
uuid: "a1e2ec51ef5d43cbb18df0696c0d159d"
format_stamp: "Formatted at 2026-09-29 16:45:22 on dist-test-slave-l31f"
server_key: "6cc5d59ebdd86d718375e75d3ed52042"
server_key_iv: "33506d4e02c0d8aced7de2559addef2a"
server_key_version: "encryptionkey@0"
I20260929 16:46:08.946784 17445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:46:08.973598 17445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:46:08.978549 17445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:46:08.978885 17445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:46:08.980051 17445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:46:08.982098 17507 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260929 16:46:08.988173 17507 ts_tablet_manager.cc:1852] T abd2bd7043154599b9ad6148bef52fe4 P a1e2ec51ef5d43cbb18df0696c0d159d: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260929 16:46:08.989246 17507 ts_tablet_manager.cc:1905] T abd2bd7043154599b9ad6148bef52fe4 P a1e2ec51ef5d43cbb18df0696c0d159d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:46:09.000165 17507 ts_tablet_manager.cc:1918] T abd2bd7043154599b9ad6148bef52fe4 P a1e2ec51ef5d43cbb18df0696c0d159d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:46:09.000438 17507 log.cc:1196] T abd2bd7043154599b9ad6148bef52fe4 P a1e2ec51ef5d43cbb18df0696c0d159d: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-0/wal/wals/abd2bd7043154599b9ad6148bef52fe4
I20260929 16:46:09.001194 17507 ts_tablet_manager.cc:1939] T abd2bd7043154599b9ad6148bef52fe4 P a1e2ec51ef5d43cbb18df0696c0d159d: Deleting consensus metadata
I20260929 16:46:09.001700 17445 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260929 16:46:09.001803 17445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.020s user 0.001s sys 0.000s
I20260929 16:46:09.001926 17445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:46:09.001991 17445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:09.042356 17445 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.65:41857
I20260929 16:46:09.042416 17613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.65:41857 every 8 connection(s)
I20260929 16:46:09.044073 17445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
I20260929 16:46:09.050217 17616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:09.047895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.65:45155)
0929 16:46:09.048916 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:09.049217 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":428,"threads_started":1}
W20260929 16:46:09.051144 17614 heartbeater.cc:644] Failed to heartbeat to 127.0.197.126:37141 (0 consecutive failures): Network error: Failed to ping master at 127.0.197.126:37141: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
I20260929 16:46:09.052397 17616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:09.051903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.65:44275)
0929 16:46:09.052099 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:09.052213 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 16:46:09.053700 17616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:09.053178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.65:47513)
0929 16:46:09.053362 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:09.053504 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260929 16:46:09.053956 17614 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 16:46:09.054419 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 17445
I20260929 16:46:09.055189 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.66:43983
--local_ip_for_outbound_sockets=127.0.197.66
--tserver_master_addrs=127.0.197.126:37141
--webserver_port=40259
--webserver_interface=127.0.197.66
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab
--principal=kudu/127.0.197.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:33025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:46:09.409875 17617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:46:09.410271 17617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:46:09.410455 17617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:46:09.419343 17617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:46:09.419533 17617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:46:09.419739 17617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.66
I20260929 16:46:09.430086 17617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:33025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab
--principal=kudu/127.0.197.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.66:43983
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.197.66
--webserver_port=40259
--tserver_master_addrs=127.0.197.126:37141
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.66
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:46:09.431591 17617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:46:09.433027 17617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:46:09.447873 17622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:09.447813 17625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:09.447916 17623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:46:09.448812 17617 server_base.cc:1034] running on GCE node
I20260929 16:46:09.449542 17617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:46:09.450690 17617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:46:09.451905 17617 hybrid_clock.cc:648] HybridClock initialized: now 1790700369451843 us; error 90 us; skew 500 ppm
I20260929 16:46:09.452365 17617 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:46:09 dist-test-slave-l31f krb5kdc[16296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:46:09.456094 17617 init.cc:377] Logged in from keytab as kudu/127.0.197.66@KRBTEST.COM (short username kudu)
I20260929 16:46:09.457818 17617 webserver.cc:460] Webserver started at http://127.0.197.66:40259/ using document root <none> and password file <none>
I20260929 16:46:09.458657 17617 fs_manager.cc:359] Metadata directory not provided
I20260929 16:46:09.458834 17617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:46:09.465248 17617 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:46:09.468678 17632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:09.470263 17617 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:46:09.470491 17617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-1/wal
uuid: "ddc41210f2dc4a3189be59eea46a2e31"
format_stamp: "Formatted at 2026-09-29 16:45:23 on dist-test-slave-l31f"
server_key: "0ecface9c95a52c3d3d9898c4d68dc2c"
server_key_iv: "8173df979e7f0113ead6a4a34c678444"
server_key_version: "encryptionkey@0"
I20260929 16:46:09.471720 17617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:46:09.501140 17617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:46:09.506063 17617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:46:09.506383 17617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:46:09.507586 17617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:46:09.509182 17617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:46:09.509325 17617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:09.509464 17617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:46:09.509552 17617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:09.546340 17617 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.66:43983
I20260929 16:46:09.546404 17744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.66:43983 every 8 connection(s)
I20260929 16:46:09.547995 17617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
I20260929 16:46:09.552490 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 17617
I20260929 16:46:09.553344 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.67:39065
--local_ip_for_outbound_sockets=127.0.197.67
--tserver_master_addrs=127.0.197.126:37141
--webserver_port=39013
--webserver_interface=127.0.197.67
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab
--principal=kudu/127.0.197.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:33025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:46:09.553964 17747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:09.551173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.66:57913)
0929 16:46:09.552192 (+ 1019us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:09.552601 (+ 409us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":398,"threads_started":1}
W20260929 16:46:09.554728 17745 heartbeater.cc:644] Failed to heartbeat to 127.0.197.126:37141 (0 consecutive failures): Network error: Failed to ping master at 127.0.197.126:37141: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
I20260929 16:46:09.555950 17747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:09.555389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.66:36007)
0929 16:46:09.555627 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:09.555758 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 16:46:09.557128 17747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:09.556634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.66:33023)
0929 16:46:09.556832 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:09.556946 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260929 16:46:09.557461 17745 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 16:46:09.910136 17748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:46:09.910454 17748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:46:09.910632 17748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:46:09.918499 17748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:46:09.918629 17748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:46:09.918872 17748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.67
I20260929 16:46:09.929360 17748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:33025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab
--principal=kudu/127.0.197.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.67:39065
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.197.67
--webserver_port=39013
--tserver_master_addrs=127.0.197.126:37141
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.67
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:46:09.930734 17748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:46:09.932204 17748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:46:09.946417 17753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:09.946475 17754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:09.946417 17756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:46:09.947726 17748 server_base.cc:1034] running on GCE node
I20260929 16:46:09.948469 17748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:46:09.949667 17748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:46:09.950919 17748 hybrid_clock.cc:648] HybridClock initialized: now 1790700369950874 us; error 43 us; skew 500 ppm
I20260929 16:46:09.951320 17748 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:46:09 dist-test-slave-l31f krb5kdc[16296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:46:09.954972 17748 init.cc:377] Logged in from keytab as kudu/127.0.197.67@KRBTEST.COM (short username kudu)
I20260929 16:46:09.956857 17748 webserver.cc:460] Webserver started at http://127.0.197.67:39013/ using document root <none> and password file <none>
I20260929 16:46:09.957665 17748 fs_manager.cc:359] Metadata directory not provided
I20260929 16:46:09.957832 17748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:46:09.963953 17748 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:46:09.967245 17763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:09.969220 17748 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 16:46:09.969466 17748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-2/wal
uuid: "ba9242ba774841c0b786fec741f15825"
format_stamp: "Formatted at 2026-09-29 16:45:23 on dist-test-slave-l31f"
server_key: "e6f6b765e1513e73a86fc5be594d7aaf"
server_key_iv: "878dbc3927fc939010fddd3e7521120a"
server_key_version: "encryptionkey@0"
I20260929 16:46:09.970602 17748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:46:09.985731 17748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:46:09.990759 17748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:46:09.991164 17748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:46:09.992329 17748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:46:09.993924 17748 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:46:09.994084 17748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:09.994221 17748 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:46:09.994311 17748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:10.029873 17748 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.67:39065
I20260929 16:46:10.029978 17875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.67:39065 every 8 connection(s)
I20260929 16:46:10.031538 17748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
I20260929 16:46:10.036722 17878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:10.034743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.67:60783)
0929 16:46:10.035572 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:10.035841 (+ 269us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":364,"threads_started":1}
W20260929 16:46:10.037526 17876 heartbeater.cc:644] Failed to heartbeat to 127.0.197.126:37141 (0 consecutive failures): Network error: Failed to ping master at 127.0.197.126:37141: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
I20260929 16:46:10.038483 17878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:10.038118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.67:33723)
0929 16:46:10.038289 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:10.038370 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 16:46:10.039528 17878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:10.039158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.67:58747)
0929 16:46:10.039298 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:10.039389 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 16:46:10.039796 17876 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 16:46:10.041630 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 17748
I20260929 16:46:10.055650 17880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:10.054639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.65:48785)
0929 16:46:10.055321 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:10.055407 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":447,"threads_started":1}
I20260929 16:46:10.559161 17881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:10.558151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.66:52783)
0929 16:46:10.558725 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:10.558834 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":268,"threads_started":1}
I20260929 16:46:11.041308 17882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:11.040417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.67:48237)
0929 16:46:11.040925 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:11.041039 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":266,"threads_started":1}
I20260929 16:46:11.057495 17883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:11.056695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.65:35349)
0929 16:46:11.057238 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:11.057332 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":311,"threads_started":1}
I20260929 16:46:11.561024 17884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:11.560225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.66:36821)
0929 16:46:11.560737 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:11.560831 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":253,"threads_started":1}
I20260929 16:46:12.043399 17885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:12.042493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.67:49101)
0929 16:46:12.043138 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:12.043233 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":268,"threads_started":1}
I20260929 16:46:12.060081 17886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:12.058768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.65:44777)
0929 16:46:12.059800 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:12.059892 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":926,"thread_start_us":327,"threads_started":1}
I20260929 16:46:12.562939 17887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:12.562074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.66:40187)
0929 16:46:12.562627 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:12.562728 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":279,"threads_started":1}
I20260929 16:46:13.045356 17888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:13.044467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.67:35343)
0929 16:46:13.045082 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:13.045185 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":326,"threads_started":1}
I20260929 16:46:13.061916 17889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:13.061114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.65:56745)
0929 16:46:13.061650 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:13.061745 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":278,"threads_started":1}
I20260929 16:46:13.564961 17890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:13.564088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.66:50457)
0929 16:46:13.564584 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:13.564699 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":230,"threads_started":1}
I20260929 16:46:14.047222 17891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:14.046310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.67:58459)
0929 16:46:14.046823 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:14.046914 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":251,"threads_started":1}
I20260929 16:46:14.063849 17892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:14.063094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.65:48541)
0929 16:46:14.063584 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:14.063670 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":245,"threads_started":1}
I20260929 16:46:14.566704 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:14.565939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.66:51705)
0929 16:46:14.566425 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:14.566513 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":236,"threads_started":1}
I20260929 16:46:15.048934 17894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:15.048148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.67:38587)
0929 16:46:15.048669 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:15.048757 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":267,"threads_started":1}
I20260929 16:46:15.065791 17895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:15.065026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.65:52865)
0929 16:46:15.065535 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:15.065627 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":238,"threads_started":1}
I20260929 16:46:15.568897 17896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:15.567892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.66:48685)
0929 16:46:15.568518 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:15.568688 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":289,"threads_started":1}
I20260929 16:46:16.051461 17897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:16.050114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.67:54675)
0929 16:46:16.050915 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:16.051182 (+ 267us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":450,"threads_started":1}
I20260929 16:46:16.068271 17898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:16.067134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.65:39931)
0929 16:46:16.067902 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:16.068018 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":326,"threads_started":1}
I20260929 16:46:16.571818 17899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:16.570251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.66:39829)
0929 16:46:16.571383 (+ 1132us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:16.571520 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":960,"thread_start_us":706,"threads_started":1}
I20260929 16:46:17.054661 17900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:17.053191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.67:59087)
0929 16:46:17.053866 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:17.054019 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":306,"threads_started":1}
I20260929 16:46:17.070412 17901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:17.069417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.65:39251)
0929 16:46:17.069931 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:17.070049 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":286,"threads_started":1}
I20260929 16:46:17.574962 17902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:17.573849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.66:51117)
0929 16:46:17.574635 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:17.574742 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":496,"threads_started":1}
I20260929 16:46:18.056656 17903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:18.055872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.67:58147)
0929 16:46:18.056387 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:18.056482 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":281,"threads_started":1}
I20260929 16:46:18.072763 17904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:18.071826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.65:55289)
0929 16:46:18.072467 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:18.072582 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":300,"threads_started":1}
I20260929 16:46:18.576810 17905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:18.575982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.66:41005)
0929 16:46:18.576535 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:18.576628 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":250,"threads_started":1}
I20260929 16:46:19.058420 17906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:19.057611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.67:46855)
0929 16:46:19.058165 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:19.058247 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":301,"threads_started":1}
I20260929 16:46:19.074775 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:19.073907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.65:36987)
0929 16:46:19.074410 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:19.074555 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":263,"threads_started":1}
I20260929 16:46:19.578667 17908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:19.577848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.66:60547)
0929 16:46:19.578374 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:19.578493 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":265,"threads_started":1}
I20260929 16:46:20.060146 17909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:20.059435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.67:37635)
0929 16:46:20.059885 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:20.059978 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":274,"threads_started":1}
I20260929 16:46:20.076766 17910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:20.075884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.65:45305)
0929 16:46:20.076479 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:20.076570 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":367,"threads_started":1}
I20260929 16:46:20.580437 17911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:20.579692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.66:39715)
0929 16:46:20.580149 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:20.580256 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":227,"threads_started":1}
I20260929 16:46:21.061966 17912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:21.061174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.67:38563)
0929 16:46:21.061697 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:21.061785 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":304,"threads_started":1}
I20260929 16:46:21.078630 17913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:21.077871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.65:48077)
0929 16:46:21.078374 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:21.078463 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":292,"threads_started":1}
I20260929 16:46:21.582326 17914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:21.581533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.66:37391)
0929 16:46:21.582029 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:21.582126 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":245,"threads_started":1}
I20260929 16:46:22.063993 17915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:22.063114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.67:49413)
0929 16:46:22.063662 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:22.063752 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":287,"threads_started":1}
I20260929 16:46:22.080523 17916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:22.079715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.65:45069)
0929 16:46:22.080215 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:22.080329 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":236,"threads_started":1}
I20260929 16:46:22.584323 17917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:22.583490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.66:45287)
0929 16:46:22.584010 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:22.584118 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":246,"threads_started":1}
I20260929 16:46:23.066064 17918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:23.065107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.67:50303)
0929 16:46:23.065638 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:23.065793 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":278,"threads_started":1}
I20260929 16:46:23.082394 17919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:23.081558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.65:48007)
0929 16:46:23.082102 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:23.082202 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":266,"threads_started":1}
I20260929 16:46:23.586359 17920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:23.585388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.66:34745)
0929 16:46:23.586000 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:23.586119 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":297,"threads_started":1}
I20260929 16:46:24.068212 17921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:24.067384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.67:34517)
0929 16:46:24.067842 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:24.067933 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":227,"threads_started":1}
I20260929 16:46:24.084709 17922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:24.083853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.65:52993)
0929 16:46:24.084394 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:24.084525 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":302,"threads_started":1}
I20260929 16:46:24.588717 17923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:24.587743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.66:44827)
0929 16:46:24.588394 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:24.588509 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":425,"threads_started":1}
I20260929 16:46:25.047881 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 17445
I20260929 16:46:25.070389 17925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:25.069529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.67:39725)
0929 16:46:25.070090 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:25.070185 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":306,"threads_started":1}
I20260929 16:46:25.074265 17924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:25.049002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38824 (local address 127.0.197.65:41857)
0929 16:46:25.049739 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:25.049754 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:25.049794 (+ 40us) server_negotiation.cc:407] Connection header received
0929 16:46:25.051726 (+ 1932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:25.051764 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:25.052205 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:25.052520 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:25.054549 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:25.056005 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:25.057331 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:25.058342 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:25.058512 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:25.067261 (+ 8749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:25.067271 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:25.067290 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:46:25.067344 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:25.070635 (+ 3291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:25.071558 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:25.071566 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:25.071574 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:25.071785 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:25.072202 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:25.072207 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:25.072209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:25.072826 (+ 617us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:25.073074 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:25.073502 (+ 428us) server_negotiation.cc:299] Negotiation successful
0929 16:46:25.073948 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":410,"threads_started":1}
I20260929 16:46:25.076184 17609 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:46:25.170132 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 17445
I20260929 16:46:25.170205 17927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:25.168905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:37141 (local address 127.0.197.65:45075)
0929 16:46:25.169511 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:25.169808 (+ 297us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.126:37141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":258,"threads_started":1}
I20260929 16:46:25.189659 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 17617
I20260929 16:46:25.209626 17928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:25.190769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57030 (local address 127.0.197.66:43983)
0929 16:46:25.191620 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:25.191640 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:25.191695 (+ 55us) server_negotiation.cc:407] Connection header received
0929 16:46:25.193229 (+ 1534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:25.193239 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:25.193719 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:25.194059 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:25.195766 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:25.197161 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:25.198571 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:25.199455 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:25.199614 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:25.202594 (+ 2980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:25.202605 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:25.202627 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:46:25.202674 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:25.206013 (+ 3339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:25.206849 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:25.206858 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:25.206865 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:25.207214 (+ 349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:25.207591 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:25.207594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:25.207596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:25.208223 (+ 627us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:25.208457 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:25.208941 (+ 484us) server_negotiation.cc:299] Negotiation successful
0929 16:46:25.209327 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":402,"threads_started":1}
I20260929 16:46:25.211328 17740 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:46:25.301234 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 17617
I20260929 16:46:25.321242 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 17748
I20260929 16:46:25.342414 17930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:25.322228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60876 (local address 127.0.197.67:39065)
0929 16:46:25.322855 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:25.322870 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:25.322964 (+ 94us) server_negotiation.cc:407] Connection header received
0929 16:46:25.324710 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:25.324722 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:25.325208 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:25.325542 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:25.327314 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:25.328651 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:25.330047 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:25.330813 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:25.330979 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:25.334095 (+ 3116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:25.334105 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:25.334124 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:46:25.334174 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:25.337426 (+ 3252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:25.338482 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:25.338495 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:25.338505 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:25.338846 (+ 341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:25.339726 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:25.339734 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:25.339737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:25.340515 (+ 778us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:25.340803 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:25.341638 (+ 835us) server_negotiation.cc:299] Negotiation successful
0929 16:46:25.342095 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":283,"threads_started":1}
I20260929 16:46:25.344198 17871 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:46:25.432762 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 17748
2026-09-29T16:46:25Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65167 ms)
[----------] 30 tests from SecurityITest (317132 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:46:25 dist-test-slave-l31f krb5kdc[17934](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:46:25 dist-test-slave-l31f krb5kdc[17934](info): set up 2 sockets
Sep 29 16:46:25 dist-test-slave-l31f krb5kdc[17934](info): commencing operation
krb5kdc: starting...
W20260929 16:46:26.559391 789 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/krb5kdc/test-user.keytab.
Sep 29 16:46:26 dist-test-slave-l31f krb5kdc[17934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:46:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:46:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.197.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
I20260929 16:46:26.835932 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:33973
--webserver_interface=127.0.197.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:32911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.126:33973
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:46:27.220919 17947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:46:27.221259 17947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:46:27.221352 17947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:46:27.229110 17947 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:46:27.229249 17947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:46:27.229350 17947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:46:27.229461 17947 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:46:27.229547 17947 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:46:27.240147 17947 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:32911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.197.126:33973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:33973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.197.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:46:27.241652 17947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:46:27.243121 17947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:46:27.256785 17952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:27.256872 17955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:27.256985 17953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:46:27.257453 17947 server_base.cc:1034] running on GCE node
I20260929 16:46:27.258404 17947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:46:27.260269 17947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:46:27.261586 17947 hybrid_clock.cc:648] HybridClock initialized: now 1790700387261526 us; error 54 us; skew 500 ppm
I20260929 16:46:27.262112 17947 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:46:27 dist-test-slave-l31f krb5kdc[17934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:46:27.265843 17947 init.cc:377] Logged in from keytab as kudu/127.0.197.126@KRBTEST.COM (short username kudu)
I20260929 16:46:27.267635 17947 webserver.cc:460] Webserver started at http://127.0.197.126:37857/ using document root <none> and password file <none>
I20260929 16:46:27.268432 17947 fs_manager.cc:359] Metadata directory not provided
I20260929 16:46:27.268711 17947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:46:27.269156 17947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:46:27.272632 17947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/master-0/data/instance:
uuid: "145c196d69a74c70bf952dc22097772e"
format_stamp: "Formatted at 2026-09-29 16:46:27 on dist-test-slave-l31f"
server_key: "47ab49a2ffaefe26fe2c208458dee767"
server_key_iv: "0439bfa1bdc5b2e7eef1949c0214d7bc"
server_key_version: "encryptionkey@0"
I20260929 16:46:27.273527 17947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/master-0/wal/instance:
uuid: "145c196d69a74c70bf952dc22097772e"
format_stamp: "Formatted at 2026-09-29 16:46:27 on dist-test-slave-l31f"
server_key: "47ab49a2ffaefe26fe2c208458dee767"
server_key_iv: "0439bfa1bdc5b2e7eef1949c0214d7bc"
server_key_version: "encryptionkey@0"
I20260929 16:46:27.279639 17947 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 16:46:27.283452 17962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:27.285075 17947 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:46:27.285305 17947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "145c196d69a74c70bf952dc22097772e"
format_stamp: "Formatted at 2026-09-29 16:46:27 on dist-test-slave-l31f"
server_key: "47ab49a2ffaefe26fe2c208458dee767"
server_key_iv: "0439bfa1bdc5b2e7eef1949c0214d7bc"
server_key_version: "encryptionkey@0"
I20260929 16:46:27.285547 17947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:46:27.301493 17947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:46:27.306829 17947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:46:27.307233 17947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:46:27.328270 17947 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.126:33973
I20260929 16:46:27.328287 18013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.126:33973 every 8 connection(s)
I20260929 16:46:27.329807 17947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:46:27.333941 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 17947
I20260929 16:46:27.334095 18014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:46:27.334203 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/master-0/wal/instance
I20260929 16:46:27.335076 789 external_mini_cluster.cc:1428] Setting key 6d816388d584d40cd4060aae72f4cd4d
I20260929 16:46:27.348620 18014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 145c196d69a74c70bf952dc22097772e: Bootstrap starting.
Sep 29 16:46:27 dist-test-slave-l31f krb5kdc[17934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:46:27.352536 18014 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 145c196d69a74c70bf952dc22097772e: Neither blocks nor log segments found. Creating new log.
I20260929 16:46:27.353857 18014 log.cc:824] T 00000000000000000000000000000000 P 145c196d69a74c70bf952dc22097772e: Log is configured to *not* fsync() on all Append() calls
I20260929 16:46:27.357597 18014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 145c196d69a74c70bf952dc22097772e: No bootstrap required, opened a new log
I20260929 16:46:27.361867 18017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:27.337384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37644 (local address 127.0.197.126:33973)
0929 16:46:27.338368 (+ 984us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:27.338381 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:27.338427 (+ 46us) server_negotiation.cc:407] Connection header received
0929 16:46:27.341150 (+ 2723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:27.341186 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:27.341818 (+ 632us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:27.342347 (+ 529us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:27.344028 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:27.345432 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:27.346925 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:27.347784 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:27.348117 (+ 333us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:27.353059 (+ 4942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:27.353077 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:27.353097 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:46:27.353153 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:27.356918 (+ 3765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:27.357799 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:27.357807 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:27.357814 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:27.358042 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:27.358497 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:27.358502 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:27.358506 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:27.359351 (+ 845us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:27.359709 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:27.360238 (+ 529us) server_negotiation.cc:299] Negotiation successful
0929 16:46:27.360756 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":756,"thread_start_us":448,"threads_started":1}
I20260929 16:46:27.368166 18014 raft_consensus.cc:348] T 00000000000000000000000000000000 P 145c196d69a74c70bf952dc22097772e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "145c196d69a74c70bf952dc22097772e" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33973 } }
I20260929 16:46:27.368636 18014 raft_consensus.cc:374] T 00000000000000000000000000000000 P 145c196d69a74c70bf952dc22097772e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:46:27.368750 18014 raft_consensus.cc:729] T 00000000000000000000000000000000 P 145c196d69a74c70bf952dc22097772e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 145c196d69a74c70bf952dc22097772e, State: Initialized, Role: FOLLOWER
I20260929 16:46:27.369415 18014 consensus_queue.cc:260] T 00000000000000000000000000000000 P 145c196d69a74c70bf952dc22097772e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "145c196d69a74c70bf952dc22097772e" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33973 } }
I20260929 16:46:27.369686 18014 raft_consensus.cc:388] T 00000000000000000000000000000000 P 145c196d69a74c70bf952dc22097772e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:46:27.369840 18014 raft_consensus.cc:482] T 00000000000000000000000000000000 P 145c196d69a74c70bf952dc22097772e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:46:27.370069 18014 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 145c196d69a74c70bf952dc22097772e [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:46:27.372843 18014 raft_consensus.cc:504] T 00000000000000000000000000000000 P 145c196d69a74c70bf952dc22097772e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "145c196d69a74c70bf952dc22097772e" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33973 } }
I20260929 16:46:27.373427 18014 leader_election.cc:302] T 00000000000000000000000000000000 P 145c196d69a74c70bf952dc22097772e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 145c196d69a74c70bf952dc22097772e; no voters:
I20260929 16:46:27.374082 18014 leader_election.cc:288] T 00000000000000000000000000000000 P 145c196d69a74c70bf952dc22097772e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:46:27.374321 18019 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 145c196d69a74c70bf952dc22097772e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:46:27.375422 18019 raft_consensus.cc:686] T 00000000000000000000000000000000 P 145c196d69a74c70bf952dc22097772e [term 1 LEADER]: Becoming Leader. State: Replica: 145c196d69a74c70bf952dc22097772e, State: Running, Role: LEADER
I20260929 16:46:27.376118 18019 consensus_queue.cc:237] T 00000000000000000000000000000000 P 145c196d69a74c70bf952dc22097772e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "145c196d69a74c70bf952dc22097772e" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33973 } }
I20260929 16:46:27.376587 18014 sys_catalog.cc:564] T 00000000000000000000000000000000 P 145c196d69a74c70bf952dc22097772e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:46:27.379266 18020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 145c196d69a74c70bf952dc22097772e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "145c196d69a74c70bf952dc22097772e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "145c196d69a74c70bf952dc22097772e" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33973 } } }
I20260929 16:46:27.379330 18021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 145c196d69a74c70bf952dc22097772e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 145c196d69a74c70bf952dc22097772e. Latest consensus state: current_term: 1 leader_uuid: "145c196d69a74c70bf952dc22097772e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "145c196d69a74c70bf952dc22097772e" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33973 } } }
I20260929 16:46:27.379719 18020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 145c196d69a74c70bf952dc22097772e [sys.catalog]: This master's current role is: LEADER
I20260929 16:46:27.379773 18021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 145c196d69a74c70bf952dc22097772e [sys.catalog]: This master's current role is: LEADER
I20260929 16:46:27.382825 18027 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:46:27.389585 18027 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:46:27.399725 18027 catalog_manager.cc:1347] Generated new cluster ID: 51c0d8ddd74f4fb5b5f62acc92261b31
I20260929 16:46:27.399881 18027 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:46:27.437054 18027 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:46:27.439299 18027 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:46:27.446638 18027 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 145c196d69a74c70bf952dc22097772e: Generated new TSK 0
I20260929 16:46:27.447799 18027 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.197.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab.
Entry for principal kudu/127.0.197.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab.
I20260929 16:46:27.516521 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.65:0
--local_ip_for_outbound_sockets=127.0.197.65
--webserver_interface=127.0.197.65
--webserver_port=0
--tserver_master_addrs=127.0.197.126:33973
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab
--principal=kudu/127.0.197.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:32911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:46:27.879228 18040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:46:27.879611 18040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:46:27.879777 18040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:46:27.888388 18040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:46:27.888584 18040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:46:27.888768 18040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.65
I20260929 16:46:27.899538 18040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:32911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab
--principal=kudu/127.0.197.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.197.65
--webserver_port=0
--tserver_master_addrs=127.0.197.126:33973
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.65
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:46:27.900967 18040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:46:27.902241 18040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:46:27.916885 18045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:27.916792 18048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:27.916834 18046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:46:27.920904 18040 server_base.cc:1034] running on GCE node
I20260929 16:46:27.921689 18040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:46:27.922792 18040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:46:27.924103 18040 hybrid_clock.cc:648] HybridClock initialized: now 1790700387924018 us; error 64 us; skew 500 ppm
I20260929 16:46:27.924547 18040 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:46:27 dist-test-slave-l31f krb5kdc[17934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:46:27.928349 18040 init.cc:377] Logged in from keytab as kudu/127.0.197.65@KRBTEST.COM (short username kudu)
I20260929 16:46:27.930109 18040 webserver.cc:460] Webserver started at http://127.0.197.65:41797/ using document root <none> and password file <none>
I20260929 16:46:27.930899 18040 fs_manager.cc:359] Metadata directory not provided
I20260929 16:46:27.931088 18040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:46:27.931439 18040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:46:27.934612 18040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/ts-0/data/instance:
uuid: "d6854e3036cd43fe915363075381c8cf"
format_stamp: "Formatted at 2026-09-29 16:46:27 on dist-test-slave-l31f"
server_key: "fde468549e0f221cd44acde8a4653f52"
server_key_iv: "80b8ea77676f020a2893ab3dabbf5277"
server_key_version: "encryptionkey@0"
I20260929 16:46:27.935591 18040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/ts-0/wal/instance:
uuid: "d6854e3036cd43fe915363075381c8cf"
format_stamp: "Formatted at 2026-09-29 16:46:27 on dist-test-slave-l31f"
server_key: "fde468549e0f221cd44acde8a4653f52"
server_key_iv: "80b8ea77676f020a2893ab3dabbf5277"
server_key_version: "encryptionkey@0"
I20260929 16:46:27.941500 18040 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260929 16:46:27.945778 18055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:27.947520 18040 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 16:46:27.947754 18040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/ts-0/wal
uuid: "d6854e3036cd43fe915363075381c8cf"
format_stamp: "Formatted at 2026-09-29 16:46:27 on dist-test-slave-l31f"
server_key: "fde468549e0f221cd44acde8a4653f52"
server_key_iv: "80b8ea77676f020a2893ab3dabbf5277"
server_key_version: "encryptionkey@0"
I20260929 16:46:27.947980 18040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:46:27.962026 18040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:46:27.966741 18040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:46:27.967170 18040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:46:27.968297 18040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:46:27.969794 18040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:46:27.969871 18040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:27.970043 18040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:46:27.970118 18040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:28.006985 18040 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.65:38127
I20260929 16:46:28.007086 18167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.65:38127 every 8 connection(s)
I20260929 16:46:28.008430 18040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
I20260929 16:46:28.017041 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 18040
I20260929 16:46:28.017290 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/ts-0/wal/instance
I20260929 16:46:28.018038 789 external_mini_cluster.cc:1428] Setting key d7ce427eb4250836fe60e7c28e4f1578
Sep 29 16:46:28 dist-test-slave-l31f krb5kdc[17934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.65@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:46:28.032979 18170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:28.011408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:40527 (local address 127.0.197.126:33973)
0929 16:46:28.012033 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:28.012038 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:28.013206 (+ 1168us) server_negotiation.cc:407] Connection header received
0929 16:46:28.015281 (+ 2075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:28.015287 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:28.015395 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:28.015527 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:28.017720 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:28.018652 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:28.021037 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:28.021470 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:28.021587 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:28.026457 (+ 4870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:28.026463 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:28.026466 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:46:28.026505 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:28.029334 (+ 2829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:28.030242 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:28.030247 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:28.030249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:28.030384 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:28.030887 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:28.030891 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:28.030893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:28.031298 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:28.031462 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:28.032506 (+ 1044us) server_negotiation.cc:299] Negotiation successful
0929 16:46:28.032703 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":382,"threads_started":1}
I20260929 16:46:28.034626 18171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:28.012179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:33973 (local address 127.0.197.65:40527)
0929 16:46:28.012981 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:28.013027 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 16:46:28.014869 (+ 1842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:46:28.015809 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:46:28.015818 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:46:28.016413 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:46:28.017419 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:28.017456 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:28.018883 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:28.018887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:28.020855 (+ 1968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:28.020874 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:28.021758 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:28.021769 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:28.022161 (+ 392us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:28.023099 (+ 938us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:46:28.023137 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:46:28.026128 (+ 2991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:46:28.029551 (+ 3423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:46:28.029562 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:46:28.029585 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:46:28.030074 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:46:28.030512 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:46:28.030517 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:46:28.030521 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:46:28.030744 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:46:28.031463 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:46:28.031474 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:46:28.032032 (+ 558us) client_negotiation.cc:769] Sending connection context
0929 16:46:28.032384 (+ 352us) client_negotiation.cc:240] Negotiation successful
0929 16:46:28.033051 (+ 667us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":321,"threads_started":1}
I20260929 16:46:28.036633 18168 heartbeater.cc:347] Connected to a master server at 127.0.197.126:33973
I20260929 16:46:28.037015 18168 heartbeater.cc:464] Registering TS with master...
I20260929 16:46:28.037976 18168 heartbeater.cc:511] Master 127.0.197.126:33973 requested a full tablet report, sending...
I20260929 16:46:28.040863 17979 ts_manager.cc:194] Registered new tserver with Master: d6854e3036cd43fe915363075381c8cf (127.0.197.65:38127)
I20260929 16:46:28.043236 17979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.65@KRBTEST.COM'} at 127.0.197.65:40527
WARNING: no policy specified for kudu/127.0.197.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab.
Entry for principal kudu/127.0.197.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab.
I20260929 16:46:28.082566 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.66:0
--local_ip_for_outbound_sockets=127.0.197.66
--webserver_interface=127.0.197.66
--webserver_port=0
--tserver_master_addrs=127.0.197.126:33973
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab
--principal=kudu/127.0.197.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:32911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:46:28.435897 18174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:46:28.436221 18174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:46:28.436400 18174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:46:28.444386 18174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:46:28.444526 18174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:46:28.444721 18174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.66
I20260929 16:46:28.455482 18174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:32911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab
--principal=kudu/127.0.197.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.197.66
--webserver_port=0
--tserver_master_addrs=127.0.197.126:33973
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.66
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:46:28.456923 18174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:46:28.458441 18174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:46:28.472878 18179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:28.472929 18180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:28.472898 18182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:46:28.474025 18174 server_base.cc:1034] running on GCE node
I20260929 16:46:28.474721 18174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:46:28.475909 18174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:46:28.477150 18174 hybrid_clock.cc:648] HybridClock initialized: now 1790700388477086 us; error 62 us; skew 500 ppm
I20260929 16:46:28.477538 18174 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:46:28 dist-test-slave-l31f krb5kdc[17934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:46:28.481446 18174 init.cc:377] Logged in from keytab as kudu/127.0.197.66@KRBTEST.COM (short username kudu)
I20260929 16:46:28.483371 18174 webserver.cc:460] Webserver started at http://127.0.197.66:45591/ using document root <none> and password file <none>
I20260929 16:46:28.484105 18174 fs_manager.cc:359] Metadata directory not provided
I20260929 16:46:28.484216 18174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:46:28.484658 18174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:46:28.487911 18174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/ts-1/data/instance:
uuid: "fcd3085f75974dbdb04998f8e84ee1ee"
format_stamp: "Formatted at 2026-09-29 16:46:28 on dist-test-slave-l31f"
server_key: "fdf43ffad4b4177112ab4d3b8d86fd5f"
server_key_iv: "495b162a9198f4f1b59f9e70cdcfc86d"
server_key_version: "encryptionkey@0"
I20260929 16:46:28.488758 18174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/ts-1/wal/instance:
uuid: "fcd3085f75974dbdb04998f8e84ee1ee"
format_stamp: "Formatted at 2026-09-29 16:46:28 on dist-test-slave-l31f"
server_key: "fdf43ffad4b4177112ab4d3b8d86fd5f"
server_key_iv: "495b162a9198f4f1b59f9e70cdcfc86d"
server_key_version: "encryptionkey@0"
I20260929 16:46:28.494489 18174 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 16:46:28.498345 18189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:28.499920 18174 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:46:28.500113 18174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/ts-1/wal
uuid: "fcd3085f75974dbdb04998f8e84ee1ee"
format_stamp: "Formatted at 2026-09-29 16:46:28 on dist-test-slave-l31f"
server_key: "fdf43ffad4b4177112ab4d3b8d86fd5f"
server_key_iv: "495b162a9198f4f1b59f9e70cdcfc86d"
server_key_version: "encryptionkey@0"
I20260929 16:46:28.500301 18174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:46:28.513559 18174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:46:28.518352 18174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:46:28.518641 18174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:46:28.519732 18174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:46:28.521157 18174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:46:28.521238 18174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:28.521329 18174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:46:28.521374 18174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:28.555965 18174 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.66:42351
I20260929 16:46:28.556067 18301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.66:42351 every 8 connection(s)
I20260929 16:46:28.557330 18174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
I20260929 16:46:28.559531 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 18174
I20260929 16:46:28.559721 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/ts-1/wal/instance
I20260929 16:46:28.560262 789 external_mini_cluster.cc:1428] Setting key d7de15d0fe9e3d5b38816711a7acd775
Sep 29 16:46:28 dist-test-slave-l31f krb5kdc[17934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.66@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:46:28.581730 18304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:28.560179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:52813 (local address 127.0.197.126:33973)
0929 16:46:28.562445 (+ 2266us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:28.562454 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:28.562481 (+ 27us) server_negotiation.cc:407] Connection header received
0929 16:46:28.564663 (+ 2182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:28.564669 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:28.564762 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:28.564886 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:28.566504 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:28.567313 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:28.568995 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:28.569421 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:28.569562 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:28.574108 (+ 4546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:28.574118 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:28.574126 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:46:28.574177 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:28.577980 (+ 3803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:28.578889 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:28.578898 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:28.578904 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:28.579108 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:28.579563 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:28.579571 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:28.579577 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:28.580037 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:28.580188 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:28.580945 (+ 757us) server_negotiation.cc:299] Negotiation successful
0929 16:46:28.581224 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2136,"thread_start_us":2035,"threads_started":1}
I20260929 16:46:28.582509 18305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:28.560919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:33973 (local address 127.0.197.66:52813)
0929 16:46:28.562003 (+ 1084us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:28.562055 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 16:46:28.564355 (+ 2300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:46:28.565151 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:46:28.565160 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:46:28.565715 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:46:28.566309 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:28.566327 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:28.567479 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:28.567483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:28.568838 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:28.568852 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:28.569671 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:28.569678 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:28.569918 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:28.570744 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:46:28.570781 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:46:28.573802 (+ 3021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:46:28.578214 (+ 4412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:46:28.578223 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:46:28.578239 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:46:28.578734 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:46:28.579230 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:46:28.579233 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:46:28.579235 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:46:28.579438 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:46:28.580170 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:46:28.580177 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:46:28.580525 (+ 348us) client_negotiation.cc:769] Sending connection context
0929 16:46:28.580777 (+ 252us) client_negotiation.cc:240] Negotiation successful
0929 16:46:28.581296 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"thread_start_us":482,"threads_started":1}
I20260929 16:46:28.584168 18302 heartbeater.cc:347] Connected to a master server at 127.0.197.126:33973
I20260929 16:46:28.584496 18302 heartbeater.cc:464] Registering TS with master...
I20260929 16:46:28.585207 18302 heartbeater.cc:511] Master 127.0.197.126:33973 requested a full tablet report, sending...
I20260929 16:46:28.586823 17979 ts_manager.cc:194] Registered new tserver with Master: fcd3085f75974dbdb04998f8e84ee1ee (127.0.197.66:42351)
I20260929 16:46:28.588121 17979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.66@KRBTEST.COM'} at 127.0.197.66:52813
WARNING: no policy specified for kudu/127.0.197.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab.
Entry for principal kudu/127.0.197.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab.
I20260929 16:46:28.623157 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.67:0
--local_ip_for_outbound_sockets=127.0.197.67
--webserver_interface=127.0.197.67
--webserver_port=0
--tserver_master_addrs=127.0.197.126:33973
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab
--principal=kudu/127.0.197.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:32911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:46:28.971896 18308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:46:28.972203 18308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:46:28.972378 18308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:46:28.980229 18308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:46:28.980368 18308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:46:28.980552 18308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.67
I20260929 16:46:28.991081 18308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:32911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab
--principal=kudu/127.0.197.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.197.67
--webserver_port=0
--tserver_master_addrs=127.0.197.126:33973
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.67
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:46:28.992478 18308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:46:28.993862 18308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:46:29.019810 18313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:29.019821 18314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:46:29.020509 18308 server_base.cc:1034] running on GCE node
W20260929 16:46:29.020699 18316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:46:29.021493 18308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:46:29.022615 18308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:46:29.023872 18308 hybrid_clock.cc:648] HybridClock initialized: now 1790700389023812 us; error 59 us; skew 500 ppm
I20260929 16:46:29.024253 18308 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:46:29 dist-test-slave-l31f krb5kdc[17934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700389, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:46:29.028129 18308 init.cc:377] Logged in from keytab as kudu/127.0.197.67@KRBTEST.COM (short username kudu)
I20260929 16:46:29.029748 18308 webserver.cc:460] Webserver started at http://127.0.197.67:36143/ using document root <none> and password file <none>
I20260929 16:46:29.030483 18308 fs_manager.cc:359] Metadata directory not provided
I20260929 16:46:29.030602 18308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:46:29.030895 18308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:46:29.034209 18308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/ts-2/data/instance:
uuid: "7074561461ff4c85b38e2fefe057cce6"
format_stamp: "Formatted at 2026-09-29 16:46:29 on dist-test-slave-l31f"
server_key: "d65646dc1a20e84b87a467b872fcc5d2"
server_key_iv: "da568c7e856654c150549ae85c94a233"
server_key_version: "encryptionkey@0"
I20260929 16:46:29.035123 18308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/ts-2/wal/instance:
uuid: "7074561461ff4c85b38e2fefe057cce6"
format_stamp: "Formatted at 2026-09-29 16:46:29 on dist-test-slave-l31f"
server_key: "d65646dc1a20e84b87a467b872fcc5d2"
server_key_iv: "da568c7e856654c150549ae85c94a233"
server_key_version: "encryptionkey@0"
I20260929 16:46:29.040738 18308 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 16:46:29.044788 18323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:29.046248 18308 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:46:29.046461 18308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/ts-2/wal
uuid: "7074561461ff4c85b38e2fefe057cce6"
format_stamp: "Formatted at 2026-09-29 16:46:29 on dist-test-slave-l31f"
server_key: "d65646dc1a20e84b87a467b872fcc5d2"
server_key_iv: "da568c7e856654c150549ae85c94a233"
server_key_version: "encryptionkey@0"
I20260929 16:46:29.046705 18308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:46:29.047701 18168 heartbeater.cc:503] Master 127.0.197.126:33973 was elected leader, sending a full tablet report...
I20260929 16:46:29.070609 18308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:46:29.075546 18308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:46:29.075879 18308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:46:29.077028 18308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:46:29.078547 18308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:46:29.078629 18308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:29.078763 18308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:46:29.078815 18308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:29.114337 18308 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.67:38181
I20260929 16:46:29.114391 18435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.67:38181 every 8 connection(s)
I20260929 16:46:29.115768 18308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
I20260929 16:46:29.125890 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 18308
I20260929 16:46:29.126104 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/ts-2/wal/instance
I20260929 16:46:29.126541 789 external_mini_cluster.cc:1428] Setting key fc7c6cf6300ac261ad8e4d9258d6eff8
Sep 29 16:46:29 dist-test-slave-l31f krb5kdc[17934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700389, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.67@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:46:29.138023 18438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:29.118541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:55505 (local address 127.0.197.126:33973)
0929 16:46:29.119094 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:29.119101 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:29.120206 (+ 1105us) server_negotiation.cc:407] Connection header received
0929 16:46:29.122157 (+ 1951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:29.122162 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:29.122255 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:29.122363 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:29.124266 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:29.125051 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:29.126835 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:29.127425 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:29.127539 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:29.131802 (+ 4263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:29.131808 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:29.131811 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:46:29.131850 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:29.134762 (+ 2912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:29.135754 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:29.135759 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:29.135762 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:29.135904 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:29.136375 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:29.136379 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:29.136381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:29.136669 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:29.136793 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:29.137544 (+ 751us) server_negotiation.cc:299] Negotiation successful
0929 16:46:29.137732 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":252,"threads_started":1}
I20260929 16:46:29.139117 18439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:29.119179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:33973 (local address 127.0.197.67:55505)
0929 16:46:29.120008 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:29.120059 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 16:46:29.121827 (+ 1768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:46:29.122658 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:46:29.122668 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:46:29.123412 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:46:29.124046 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:29.124067 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:29.125222 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:29.125226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:29.126621 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:29.126662 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:29.127670 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:29.127680 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:29.127943 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:29.128822 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:46:29.128879 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:46:29.131555 (+ 2676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:46:29.134953 (+ 3398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:46:29.134962 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:46:29.134977 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:46:29.135588 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:46:29.136043 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:46:29.136047 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:46:29.136050 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:46:29.136256 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:46:29.136804 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:46:29.136811 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:46:29.137195 (+ 384us) client_negotiation.cc:769] Sending connection context
0929 16:46:29.137459 (+ 264us) client_negotiation.cc:240] Negotiation successful
0929 16:46:29.137876 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":329,"threads_started":1}
I20260929 16:46:29.140772 18436 heartbeater.cc:347] Connected to a master server at 127.0.197.126:33973
I20260929 16:46:29.141207 18436 heartbeater.cc:464] Registering TS with master...
I20260929 16:46:29.142172 18436 heartbeater.cc:511] Master 127.0.197.126:33973 requested a full tablet report, sending...
I20260929 16:46:29.143812 17979 ts_manager.cc:194] Registered new tserver with Master: 7074561461ff4c85b38e2fefe057cce6 (127.0.197.67:38181)
I20260929 16:46:29.144613 17979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.67@KRBTEST.COM'} at 127.0.197.67:55505
I20260929 16:46:29.156205 789 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:46:29.174969 18438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:29.161298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37658 (local address 127.0.197.126:33973)
0929 16:46:29.161484 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:29.161488 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:29.161872 (+ 384us) server_negotiation.cc:407] Connection header received
0929 16:46:29.162029 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:29.162033 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:29.162116 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:29.162210 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:29.163798 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:29.164554 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:29.165976 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:29.166387 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:29.166498 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:29.169355 (+ 2857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:29.169360 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:29.169363 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:46:29.169402 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:29.172431 (+ 3029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:29.173136 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:29.173141 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:29.173143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:29.173273 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:29.173652 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:29.173656 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:29.173659 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:29.174023 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:29.174234 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:29.174555 (+ 321us) server_negotiation.cc:299] Negotiation successful
0929 16:46:29.174742 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 16:46:29.179739 17979 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37658:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:46:29.187474 17979 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:46:29.217274 18450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:29.201287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48890 (local address 127.0.197.67:38181)
0929 16:46:29.205189 (+ 3902us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:29.205197 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:29.205234 (+ 37us) server_negotiation.cc:407] Connection header received
0929 16:46:29.205347 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:29.205354 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:29.205620 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:29.205857 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:46:29.207064 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:29.208766 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:29.212823 (+ 4057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:29.214887 (+ 2064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:29.215121 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:29.215483 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:29.216701 (+ 1218us) server_negotiation.cc:299] Negotiation successful
0929 16:46:29.216920 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3734,"thread_start_us":3588,"threads_started":1}
I20260929 16:46:29.217270 18451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:29.201016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.67:38181 (local address 127.0.0.1:48890)
0929 16:46:29.202400 (+ 1384us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:29.202448 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 16:46:29.202620 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:46:29.206194 (+ 3574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:46:29.206204 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:46:29.206236 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:46:29.206766 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:29.206784 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:29.208947 (+ 2163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:29.208957 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:29.212566 (+ 3609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:29.212596 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:29.215707 (+ 3111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:29.215717 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:29.216104 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:29.216192 (+ 88us) client_negotiation.cc:769] Sending connection context
0929 16:46:29.216384 (+ 192us) client_negotiation.cc:240] Negotiation successful
0929 16:46:29.216724 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1111,"thread_start_us":4364,"threads_started":1}
I20260929 16:46:29.217572 18452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:29.202023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51600 (local address 127.0.197.65:38127)
0929 16:46:29.204687 (+ 2664us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:29.204696 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:29.205978 (+ 1282us) server_negotiation.cc:407] Connection header received
0929 16:46:29.206134 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:29.206145 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:29.206433 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:29.206674 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:46:29.207936 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:29.210094 (+ 2158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:29.213773 (+ 3679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:29.215919 (+ 2146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:29.216044 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:29.216299 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:29.216943 (+ 644us) server_negotiation.cc:299] Negotiation successful
0929 16:46:29.217140 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2453,"thread_start_us":3176,"threads_started":1}
I20260929 16:46:29.218369 18449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:29.201467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:38127 (local address 127.0.0.1:51600)
0929 16:46:29.205647 (+ 4180us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:29.205669 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:46:29.205807 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:46:29.207041 (+ 1234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:46:29.207050 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:46:29.207089 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:46:29.207656 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:29.207674 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:29.210357 (+ 2683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:29.210367 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:29.213497 (+ 3130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:29.213522 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:29.216507 (+ 2985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:29.216512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:29.216720 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:29.216750 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 16:46:29.217848 (+ 1098us) client_negotiation.cc:240] Negotiation successful
0929 16:46:29.218021 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4045,"thread_start_us":701,"threads_started":1}
I20260929 16:46:29.221913 18103 tablet_service.cc:1467] Processing CreateTablet for tablet 24eae579092b4574858b0da33fc0ad73 (DEFAULT_TABLE table=test-table [id=588dbbd68e8f4b3a96e48ea0b22aeca0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:46:29.221913 18371 tablet_service.cc:1467] Processing CreateTablet for tablet 24eae579092b4574858b0da33fc0ad73 (DEFAULT_TABLE table=test-table [id=588dbbd68e8f4b3a96e48ea0b22aeca0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:46:29.223650 18103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24eae579092b4574858b0da33fc0ad73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:46:29.223673 18371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24eae579092b4574858b0da33fc0ad73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:46:29.224917 18454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:29.206124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60564 (local address 127.0.197.66:42351)
0929 16:46:29.217967 (+ 11843us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:29.217976 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:29.218009 (+ 33us) server_negotiation.cc:407] Connection header received
0929 16:46:29.218120 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:29.218132 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:29.218442 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:29.218663 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:46:29.219567 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:29.220715 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:29.222412 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:29.223627 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:29.223811 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:29.224035 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:29.224380 (+ 345us) server_negotiation.cc:299] Negotiation successful
0929 16:46:29.224566 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11658,"thread_start_us":13540,"threads_started":1}
I20260929 16:46:29.225435 18453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:29.205852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.66:42351 (local address 127.0.0.1:60564)
0929 16:46:29.216108 (+ 10256us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:29.216130 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:46:29.216264 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:46:29.218923 (+ 2659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:46:29.218927 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:46:29.218956 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:46:29.219346 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:29.219361 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:29.220875 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:29.220881 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:29.222220 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:29.222233 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:29.224002 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:29.224009 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:29.224175 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:29.224203 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 16:46:29.224327 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 16:46:29.224483 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10096,"thread_start_us":10514,"threads_started":1}
I20260929 16:46:29.228974 18237 tablet_service.cc:1467] Processing CreateTablet for tablet 24eae579092b4574858b0da33fc0ad73 (DEFAULT_TABLE table=test-table [id=588dbbd68e8f4b3a96e48ea0b22aeca0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:46:29.230363 18237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24eae579092b4574858b0da33fc0ad73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:46:29.240111 18455 tablet_bootstrap.cc:492] T 24eae579092b4574858b0da33fc0ad73 P d6854e3036cd43fe915363075381c8cf: Bootstrap starting.
I20260929 16:46:29.241003 18456 tablet_bootstrap.cc:492] T 24eae579092b4574858b0da33fc0ad73 P 7074561461ff4c85b38e2fefe057cce6: Bootstrap starting.
I20260929 16:46:29.244053 18457 tablet_bootstrap.cc:492] T 24eae579092b4574858b0da33fc0ad73 P fcd3085f75974dbdb04998f8e84ee1ee: Bootstrap starting.
I20260929 16:46:29.245713 18455 tablet_bootstrap.cc:654] T 24eae579092b4574858b0da33fc0ad73 P d6854e3036cd43fe915363075381c8cf: Neither blocks nor log segments found. Creating new log.
I20260929 16:46:29.245980 18456 tablet_bootstrap.cc:654] T 24eae579092b4574858b0da33fc0ad73 P 7074561461ff4c85b38e2fefe057cce6: Neither blocks nor log segments found. Creating new log.
I20260929 16:46:29.246963 18455 log.cc:824] T 24eae579092b4574858b0da33fc0ad73 P d6854e3036cd43fe915363075381c8cf: Log is configured to *not* fsync() on all Append() calls
I20260929 16:46:29.247289 18456 log.cc:824] T 24eae579092b4574858b0da33fc0ad73 P 7074561461ff4c85b38e2fefe057cce6: Log is configured to *not* fsync() on all Append() calls
I20260929 16:46:29.247805 18457 tablet_bootstrap.cc:654] T 24eae579092b4574858b0da33fc0ad73 P fcd3085f75974dbdb04998f8e84ee1ee: Neither blocks nor log segments found. Creating new log.
I20260929 16:46:29.248960 18457 log.cc:824] T 24eae579092b4574858b0da33fc0ad73 P fcd3085f75974dbdb04998f8e84ee1ee: Log is configured to *not* fsync() on all Append() calls
I20260929 16:46:29.250270 18455 tablet_bootstrap.cc:492] T 24eae579092b4574858b0da33fc0ad73 P d6854e3036cd43fe915363075381c8cf: No bootstrap required, opened a new log
I20260929 16:46:29.250624 18455 ts_tablet_manager.cc:1397] T 24eae579092b4574858b0da33fc0ad73 P d6854e3036cd43fe915363075381c8cf: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260929 16:46:29.252337 18456 tablet_bootstrap.cc:492] T 24eae579092b4574858b0da33fc0ad73 P 7074561461ff4c85b38e2fefe057cce6: No bootstrap required, opened a new log
I20260929 16:46:29.252638 18456 ts_tablet_manager.cc:1397] T 24eae579092b4574858b0da33fc0ad73 P 7074561461ff4c85b38e2fefe057cce6: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.003s
I20260929 16:46:29.252697 18457 tablet_bootstrap.cc:492] T 24eae579092b4574858b0da33fc0ad73 P fcd3085f75974dbdb04998f8e84ee1ee: No bootstrap required, opened a new log
I20260929 16:46:29.252967 18457 ts_tablet_manager.cc:1397] T 24eae579092b4574858b0da33fc0ad73 P fcd3085f75974dbdb04998f8e84ee1ee: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260929 16:46:29.259784 18455 raft_consensus.cc:348] T 24eae579092b4574858b0da33fc0ad73 P d6854e3036cd43fe915363075381c8cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7074561461ff4c85b38e2fefe057cce6" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 38181 } } peers { permanent_uuid: "fcd3085f75974dbdb04998f8e84ee1ee" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42351 } } peers { permanent_uuid: "d6854e3036cd43fe915363075381c8cf" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 38127 } }
I20260929 16:46:29.260228 18455 raft_consensus.cc:374] T 24eae579092b4574858b0da33fc0ad73 P d6854e3036cd43fe915363075381c8cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:46:29.260332 18455 raft_consensus.cc:729] T 24eae579092b4574858b0da33fc0ad73 P d6854e3036cd43fe915363075381c8cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6854e3036cd43fe915363075381c8cf, State: Initialized, Role: FOLLOWER
I20260929 16:46:29.261008 18455 consensus_queue.cc:260] T 24eae579092b4574858b0da33fc0ad73 P d6854e3036cd43fe915363075381c8cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7074561461ff4c85b38e2fefe057cce6" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 38181 } } peers { permanent_uuid: "fcd3085f75974dbdb04998f8e84ee1ee" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42351 } } peers { permanent_uuid: "d6854e3036cd43fe915363075381c8cf" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 38127 } }
I20260929 16:46:29.264595 18455 ts_tablet_manager.cc:1428] T 24eae579092b4574858b0da33fc0ad73 P d6854e3036cd43fe915363075381c8cf: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
I20260929 16:46:29.264528 18457 raft_consensus.cc:348] T 24eae579092b4574858b0da33fc0ad73 P fcd3085f75974dbdb04998f8e84ee1ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7074561461ff4c85b38e2fefe057cce6" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 38181 } } peers { permanent_uuid: "fcd3085f75974dbdb04998f8e84ee1ee" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42351 } } peers { permanent_uuid: "d6854e3036cd43fe915363075381c8cf" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 38127 } }
I20260929 16:46:29.264528 18456 raft_consensus.cc:348] T 24eae579092b4574858b0da33fc0ad73 P 7074561461ff4c85b38e2fefe057cce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7074561461ff4c85b38e2fefe057cce6" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 38181 } } peers { permanent_uuid: "fcd3085f75974dbdb04998f8e84ee1ee" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42351 } } peers { permanent_uuid: "d6854e3036cd43fe915363075381c8cf" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 38127 } }
I20260929 16:46:29.265021 18456 raft_consensus.cc:374] T 24eae579092b4574858b0da33fc0ad73 P 7074561461ff4c85b38e2fefe057cce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:46:29.265020 18457 raft_consensus.cc:374] T 24eae579092b4574858b0da33fc0ad73 P fcd3085f75974dbdb04998f8e84ee1ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:46:29.265131 18456 raft_consensus.cc:729] T 24eae579092b4574858b0da33fc0ad73 P 7074561461ff4c85b38e2fefe057cce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7074561461ff4c85b38e2fefe057cce6, State: Initialized, Role: FOLLOWER
I20260929 16:46:29.265131 18457 raft_consensus.cc:729] T 24eae579092b4574858b0da33fc0ad73 P fcd3085f75974dbdb04998f8e84ee1ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcd3085f75974dbdb04998f8e84ee1ee, State: Initialized, Role: FOLLOWER
I20260929 16:46:29.265807 18457 consensus_queue.cc:260] T 24eae579092b4574858b0da33fc0ad73 P fcd3085f75974dbdb04998f8e84ee1ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7074561461ff4c85b38e2fefe057cce6" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 38181 } } peers { permanent_uuid: "fcd3085f75974dbdb04998f8e84ee1ee" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42351 } } peers { permanent_uuid: "d6854e3036cd43fe915363075381c8cf" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 38127 } }
I20260929 16:46:29.265812 18456 consensus_queue.cc:260] T 24eae579092b4574858b0da33fc0ad73 P 7074561461ff4c85b38e2fefe057cce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7074561461ff4c85b38e2fefe057cce6" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 38181 } } peers { permanent_uuid: "fcd3085f75974dbdb04998f8e84ee1ee" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42351 } } peers { permanent_uuid: "d6854e3036cd43fe915363075381c8cf" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 38127 } }
I20260929 16:46:29.267915 18456 ts_tablet_manager.cc:1428] T 24eae579092b4574858b0da33fc0ad73 P 7074561461ff4c85b38e2fefe057cce6: Time spent starting tablet: real 0.015s user 0.015s sys 0.001s
I20260929 16:46:29.268615 18436 heartbeater.cc:503] Master 127.0.197.126:33973 was elected leader, sending a full tablet report...
I20260929 16:46:29.268983 18457 ts_tablet_manager.cc:1428] T 24eae579092b4574858b0da33fc0ad73 P fcd3085f75974dbdb04998f8e84ee1ee: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260929 16:46:29.270195 18302 heartbeater.cc:503] Master 127.0.197.126:33973 was elected leader, sending a full tablet report...
I20260929 16:46:29.294098 18461 raft_consensus.cc:482] T 24eae579092b4574858b0da33fc0ad73 P d6854e3036cd43fe915363075381c8cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:46:29.294492 18461 raft_consensus.cc:504] T 24eae579092b4574858b0da33fc0ad73 P d6854e3036cd43fe915363075381c8cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7074561461ff4c85b38e2fefe057cce6" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 38181 } } peers { permanent_uuid: "fcd3085f75974dbdb04998f8e84ee1ee" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42351 } } peers { permanent_uuid: "d6854e3036cd43fe915363075381c8cf" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 38127 } }
I20260929 16:46:29.296342 18461 leader_election.cc:288] T 24eae579092b4574858b0da33fc0ad73 P d6854e3036cd43fe915363075381c8cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7074561461ff4c85b38e2fefe057cce6 (127.0.197.67:38181), fcd3085f75974dbdb04998f8e84ee1ee (127.0.197.66:42351)
I20260929 16:46:29.304481 18450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:29.296919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:57293 (local address 127.0.197.67:38181)
0929 16:46:29.297195 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:29.297202 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:29.297547 (+ 345us) server_negotiation.cc:407] Connection header received
0929 16:46:29.297856 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:29.297865 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:29.297956 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:29.298066 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:46:29.298947 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:29.300000 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:29.302264 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:29.303249 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:29.303467 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:29.303554 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:29.303977 (+ 423us) server_negotiation.cc:299] Negotiation successful
0929 16:46:29.304207 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 16:46:29.304534 18454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:29.297940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:55477 (local address 127.0.197.66:42351)
0929 16:46:29.298159 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:29.298165 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:29.298485 (+ 320us) server_negotiation.cc:407] Connection header received
0929 16:46:29.298627 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:29.298636 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:29.298720 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:29.298832 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:46:29.299656 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:29.300508 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:29.302438 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:29.303428 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:29.303606 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:29.303740 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:29.304127 (+ 387us) server_negotiation.cc:299] Negotiation successful
0929 16:46:29.304286 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 16:46:29.304541 18464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:29.296777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.67:38181 (local address 127.0.197.65:57293)
0929 16:46:29.297415 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:29.297434 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:46:29.297618 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:46:29.298263 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:46:29.298267 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:46:29.298295 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:46:29.298757 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:29.298773 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:29.300342 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:29.300348 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:29.302047 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:29.302062 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:29.303484 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:29.303491 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:29.303732 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:29.303782 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 16:46:29.303917 (+ 135us) client_negotiation.cc:240] Negotiation successful
0929 16:46:29.304089 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"mutex_wait_us":2,"spinlock_wait_cycles":43904,"thread_start_us":363,"threads_started":1}
I20260929 16:46:29.305310 18465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:29.297643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.66:42351 (local address 127.0.197.65:55477)
0929 16:46:29.298360 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:29.298375 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:46:29.298501 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:46:29.298954 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:46:29.298959 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:46:29.298987 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:46:29.299438 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:29.299451 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:29.300713 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:29.300717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:29.302250 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:29.302266 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:29.303732 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:29.303738 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:29.303937 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:29.303966 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 16:46:29.304738 (+ 772us) client_negotiation.cc:240] Negotiation successful
0929 16:46:29.304908 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":327,"threads_started":1}
I20260929 16:46:29.306267 18391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eae579092b4574858b0da33fc0ad73" candidate_uuid: "d6854e3036cd43fe915363075381c8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7074561461ff4c85b38e2fefe057cce6" is_pre_election: true
I20260929 16:46:29.306417 18257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eae579092b4574858b0da33fc0ad73" candidate_uuid: "d6854e3036cd43fe915363075381c8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcd3085f75974dbdb04998f8e84ee1ee" is_pre_election: true
I20260929 16:46:29.306957 18391 raft_consensus.cc:2454] T 24eae579092b4574858b0da33fc0ad73 P 7074561461ff4c85b38e2fefe057cce6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6854e3036cd43fe915363075381c8cf in term 0.
I20260929 16:46:29.306957 18257 raft_consensus.cc:2454] T 24eae579092b4574858b0da33fc0ad73 P fcd3085f75974dbdb04998f8e84ee1ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6854e3036cd43fe915363075381c8cf in term 0.
I20260929 16:46:29.307928 18058 leader_election.cc:302] T 24eae579092b4574858b0da33fc0ad73 P d6854e3036cd43fe915363075381c8cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7074561461ff4c85b38e2fefe057cce6, d6854e3036cd43fe915363075381c8cf; no voters:
I20260929 16:46:29.308467 18461 raft_consensus.cc:2781] T 24eae579092b4574858b0da33fc0ad73 P d6854e3036cd43fe915363075381c8cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:46:29.308617 18461 raft_consensus.cc:482] T 24eae579092b4574858b0da33fc0ad73 P d6854e3036cd43fe915363075381c8cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:46:29.308784 18461 raft_consensus.cc:3037] T 24eae579092b4574858b0da33fc0ad73 P d6854e3036cd43fe915363075381c8cf [term 0 FOLLOWER]: Advancing to term 1
W20260929 16:46:29.309523 18303 tablet.cc:2369] T 24eae579092b4574858b0da33fc0ad73 P fcd3085f75974dbdb04998f8e84ee1ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:46:29.311196 18461 raft_consensus.cc:504] T 24eae579092b4574858b0da33fc0ad73 P d6854e3036cd43fe915363075381c8cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7074561461ff4c85b38e2fefe057cce6" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 38181 } } peers { permanent_uuid: "fcd3085f75974dbdb04998f8e84ee1ee" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42351 } } peers { permanent_uuid: "d6854e3036cd43fe915363075381c8cf" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 38127 } }
I20260929 16:46:29.311854 18461 leader_election.cc:288] T 24eae579092b4574858b0da33fc0ad73 P d6854e3036cd43fe915363075381c8cf [CANDIDATE]: Term 1 election: Requested vote from peers 7074561461ff4c85b38e2fefe057cce6 (127.0.197.67:38181), fcd3085f75974dbdb04998f8e84ee1ee (127.0.197.66:42351)
I20260929 16:46:29.312641 18257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eae579092b4574858b0da33fc0ad73" candidate_uuid: "d6854e3036cd43fe915363075381c8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcd3085f75974dbdb04998f8e84ee1ee"
I20260929 16:46:29.312660 18391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24eae579092b4574858b0da33fc0ad73" candidate_uuid: "d6854e3036cd43fe915363075381c8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7074561461ff4c85b38e2fefe057cce6"
I20260929 16:46:29.312863 18257 raft_consensus.cc:3037] T 24eae579092b4574858b0da33fc0ad73 P fcd3085f75974dbdb04998f8e84ee1ee [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:46:29.312937 18391 raft_consensus.cc:3037] T 24eae579092b4574858b0da33fc0ad73 P 7074561461ff4c85b38e2fefe057cce6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:46:29.315413 18257 raft_consensus.cc:2454] T 24eae579092b4574858b0da33fc0ad73 P fcd3085f75974dbdb04998f8e84ee1ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6854e3036cd43fe915363075381c8cf in term 1.
I20260929 16:46:29.315516 18391 raft_consensus.cc:2454] T 24eae579092b4574858b0da33fc0ad73 P 7074561461ff4c85b38e2fefe057cce6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6854e3036cd43fe915363075381c8cf in term 1.
I20260929 16:46:29.315950 18058 leader_election.cc:302] T 24eae579092b4574858b0da33fc0ad73 P d6854e3036cd43fe915363075381c8cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6854e3036cd43fe915363075381c8cf, fcd3085f75974dbdb04998f8e84ee1ee; no voters:
I20260929 16:46:29.316375 18461 raft_consensus.cc:2781] T 24eae579092b4574858b0da33fc0ad73 P d6854e3036cd43fe915363075381c8cf [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:46:29.316891 18461 raft_consensus.cc:686] T 24eae579092b4574858b0da33fc0ad73 P d6854e3036cd43fe915363075381c8cf [term 1 LEADER]: Becoming Leader. State: Replica: d6854e3036cd43fe915363075381c8cf, State: Running, Role: LEADER
I20260929 16:46:29.317440 18461 consensus_queue.cc:237] T 24eae579092b4574858b0da33fc0ad73 P d6854e3036cd43fe915363075381c8cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7074561461ff4c85b38e2fefe057cce6" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 38181 } } peers { permanent_uuid: "fcd3085f75974dbdb04998f8e84ee1ee" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42351 } } peers { permanent_uuid: "d6854e3036cd43fe915363075381c8cf" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 38127 } }
I20260929 16:46:29.322535 17978 catalog_manager.cc:5495] T 24eae579092b4574858b0da33fc0ad73 P d6854e3036cd43fe915363075381c8cf reported cstate change: term changed from 0 to 1, leader changed from <none> to d6854e3036cd43fe915363075381c8cf (127.0.197.65). New cstate: current_term: 1 leader_uuid: "d6854e3036cd43fe915363075381c8cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7074561461ff4c85b38e2fefe057cce6" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 38181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fcd3085f75974dbdb04998f8e84ee1ee" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6854e3036cd43fe915363075381c8cf" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 38127 } health_report { overall_health: HEALTHY } } }
I20260929 16:46:29.344012 18452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:29.337387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51608 (local address 127.0.197.65:38127)
0929 16:46:29.337579 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:29.337584 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:29.337603 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:46:29.337699 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:29.337703 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:29.337785 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:29.337924 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:46:29.338404 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:29.339262 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:29.341386 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:29.341837 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:29.341954 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:29.342578 (+ 624us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:46:29.343286 (+ 708us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:46:29.343413 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:29.343544 (+ 131us) server_negotiation.cc:299] Negotiation successful
0929 16:46:29.343681 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 16:46:29.353101 18391 raft_consensus.cc:1261] T 24eae579092b4574858b0da33fc0ad73 P 7074561461ff4c85b38e2fefe057cce6 [term 1 FOLLOWER]: Refusing update from remote peer d6854e3036cd43fe915363075381c8cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:46:29.353107 18257 raft_consensus.cc:1261] T 24eae579092b4574858b0da33fc0ad73 P fcd3085f75974dbdb04998f8e84ee1ee [term 1 FOLLOWER]: Refusing update from remote peer d6854e3036cd43fe915363075381c8cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:46:29.354146 18466 consensus_queue.cc:1035] T 24eae579092b4574858b0da33fc0ad73 P d6854e3036cd43fe915363075381c8cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcd3085f75974dbdb04998f8e84ee1ee" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:46:29.354578 18461 consensus_queue.cc:1035] T 24eae579092b4574858b0da33fc0ad73 P d6854e3036cd43fe915363075381c8cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7074561461ff4c85b38e2fefe057cce6" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 38181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 16:46:29.368880 18437 tablet.cc:2369] T 24eae579092b4574858b0da33fc0ad73 P 7074561461ff4c85b38e2fefe057cce6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:46:29.377696 18470 mvcc.cc:205] Tried to move back new op lower bound from 7334708794778279936 to 7334708794647953408. Current Snapshot: MvccSnapshot[applied={T|T < 7334708794778279936 or (T in {7334708794778279936})}]
I20260929 16:46:29.384343 17978 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:46:29.384647 17978 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:46:29.388708 17978 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 145c196d69a74c70bf952dc22097772e: Sending DeleteTablet for 3 replicas of tablet 24eae579092b4574858b0da33fc0ad73
I20260929 16:46:29.390142 18103 tablet_service.cc:1514] Processing DeleteTablet for tablet 24eae579092b4574858b0da33fc0ad73 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:46:29 UTC) from {username='kudu', principal='kudu/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:51600
I20260929 16:46:29.390199 18371 tablet_service.cc:1514] Processing DeleteTablet for tablet 24eae579092b4574858b0da33fc0ad73 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:46:29 UTC) from {username='kudu', principal='kudu/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:48890
I20260929 16:46:29.390245 18237 tablet_service.cc:1514] Processing DeleteTablet for tablet 24eae579092b4574858b0da33fc0ad73 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:46:29 UTC) from {username='kudu', principal='kudu/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:60564
I20260929 16:46:29.391237 18479 tablet_replica.cc:330] T 24eae579092b4574858b0da33fc0ad73 P d6854e3036cd43fe915363075381c8cf: stopping tablet replica
I20260929 16:46:29.392035 18480 tablet_replica.cc:330] T 24eae579092b4574858b0da33fc0ad73 P 7074561461ff4c85b38e2fefe057cce6: stopping tablet replica
I20260929 16:46:29.392716 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 18040
I20260929 16:46:29.392520 18480 raft_consensus.cc:2229] T 24eae579092b4574858b0da33fc0ad73 P 7074561461ff4c85b38e2fefe057cce6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:46:29.394164 18480 raft_consensus.cc:2258] T 24eae579092b4574858b0da33fc0ad73 P 7074561461ff4c85b38e2fefe057cce6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:46:29.395025 18481 tablet_replica.cc:330] T 24eae579092b4574858b0da33fc0ad73 P fcd3085f75974dbdb04998f8e84ee1ee: stopping tablet replica
I20260929 16:46:29.395605 18481 raft_consensus.cc:2229] T 24eae579092b4574858b0da33fc0ad73 P fcd3085f75974dbdb04998f8e84ee1ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:46:29.396250 18481 raft_consensus.cc:2258] T 24eae579092b4574858b0da33fc0ad73 P fcd3085f75974dbdb04998f8e84ee1ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:46:29.399161 18480 ts_tablet_manager.cc:1905] T 24eae579092b4574858b0da33fc0ad73 P 7074561461ff4c85b38e2fefe057cce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:46:29.399634 18479 raft_consensus.cc:2229] T 24eae579092b4574858b0da33fc0ad73 P d6854e3036cd43fe915363075381c8cf [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:46:29.400458 18479 raft_consensus.cc:2258] T 24eae579092b4574858b0da33fc0ad73 P d6854e3036cd43fe915363075381c8cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:46:29.403066 18479 ts_tablet_manager.cc:1905] T 24eae579092b4574858b0da33fc0ad73 P d6854e3036cd43fe915363075381c8cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:46:29.405699 18481 ts_tablet_manager.cc:1905] T 24eae579092b4574858b0da33fc0ad73 P fcd3085f75974dbdb04998f8e84ee1ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:46:29.406884 18480 ts_tablet_manager.cc:1918] T 24eae579092b4574858b0da33fc0ad73 P 7074561461ff4c85b38e2fefe057cce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:46:29.407091 18480 log.cc:1196] T 24eae579092b4574858b0da33fc0ad73 P 7074561461ff4c85b38e2fefe057cce6: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/ts-2/wal/wals/24eae579092b4574858b0da33fc0ad73
I20260929 16:46:29.407642 18480 ts_tablet_manager.cc:1939] T 24eae579092b4574858b0da33fc0ad73 P 7074561461ff4c85b38e2fefe057cce6: Deleting consensus metadata
I20260929 16:46:29.409706 17965 catalog_manager.cc:4841] TS 7074561461ff4c85b38e2fefe057cce6 (127.0.197.67:38181): tablet 24eae579092b4574858b0da33fc0ad73 (table test-table [id=588dbbd68e8f4b3a96e48ea0b22aeca0]) successfully deleted
Sep 29 16:46:29 dist-test-slave-l31f krb5kdc[17934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.65@KRBTEST.COM
I20260929 16:46:29.416093 18481 ts_tablet_manager.cc:1918] T 24eae579092b4574858b0da33fc0ad73 P fcd3085f75974dbdb04998f8e84ee1ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:46:29.416115 18479 ts_tablet_manager.cc:1918] T 24eae579092b4574858b0da33fc0ad73 P d6854e3036cd43fe915363075381c8cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:46:29.416321 18479 log.cc:1196] T 24eae579092b4574858b0da33fc0ad73 P d6854e3036cd43fe915363075381c8cf: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/ts-0/wal/wals/24eae579092b4574858b0da33fc0ad73
I20260929 16:46:29.416333 18481 log.cc:1196] T 24eae579092b4574858b0da33fc0ad73 P fcd3085f75974dbdb04998f8e84ee1ee: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790700068255611-789-0/minicluster-data/ts-1/wal/wals/24eae579092b4574858b0da33fc0ad73
I20260929 16:46:29.416934 18481 ts_tablet_manager.cc:1939] T 24eae579092b4574858b0da33fc0ad73 P fcd3085f75974dbdb04998f8e84ee1ee: Deleting consensus metadata
I20260929 16:46:29.416967 18479 ts_tablet_manager.cc:1939] T 24eae579092b4574858b0da33fc0ad73 P d6854e3036cd43fe915363075381c8cf: Deleting consensus metadata
I20260929 16:46:29.419024 17965 catalog_manager.cc:4841] TS fcd3085f75974dbdb04998f8e84ee1ee (127.0.197.66:42351): tablet 24eae579092b4574858b0da33fc0ad73 (table test-table [id=588dbbd68e8f4b3a96e48ea0b22aeca0]) successfully deleted
I20260929 16:46:29.419607 17966 catalog_manager.cc:4841] TS d6854e3036cd43fe915363075381c8cf (127.0.197.65:38127): tablet 24eae579092b4574858b0da33fc0ad73 (table test-table [id=588dbbd68e8f4b3a96e48ea0b22aeca0]) successfully deleted
I20260929 16:46:29.424669 18452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:29.394807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51620 (local address 127.0.197.65:38127)
0929 16:46:29.395052 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:29.395058 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:29.401466 (+ 6408us) server_negotiation.cc:407] Connection header received
0929 16:46:29.401816 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:29.401822 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:29.401914 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:29.402004 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:29.404160 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:29.405398 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:29.407934 (+ 2536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:29.409968 (+ 2034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:29.410120 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:29.416293 (+ 6173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:29.416314 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:29.416338 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:46:29.416397 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:29.421059 (+ 4662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:29.422153 (+ 1094us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:29.422166 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:29.422177 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:29.422415 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:29.422966 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:29.422971 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:29.422975 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:29.423510 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:29.423741 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:29.424116 (+ 375us) server_negotiation.cc:299] Negotiation successful
0929 16:46:29.424319 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 16:46:29.425410 18163 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:46:29.529042 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 18040
I20260929 16:46:29.548672 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 18174
Sep 29 16:46:29 dist-test-slave-l31f krb5kdc[17934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.66@KRBTEST.COM
I20260929 16:46:29.565977 18454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:29.549557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60578 (local address 127.0.197.66:42351)
0929 16:46:29.549744 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:29.549749 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:29.549767 (+ 18us) server_negotiation.cc:407] Connection header received
0929 16:46:29.549825 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:29.549829 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:29.549890 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:29.549977 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:29.551490 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:29.552257 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:29.553636 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:29.554012 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:29.554143 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:29.559174 (+ 5031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:29.559189 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:29.559202 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:46:29.559245 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:29.562515 (+ 3270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:29.563530 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:29.563542 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:29.563546 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:29.563761 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:29.564328 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:29.564332 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:29.564335 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:29.564789 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:29.565045 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:29.565428 (+ 383us) server_negotiation.cc:299] Negotiation successful
0929 16:46:29.565620 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 16:46:29.566702 18297 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:46:29.660964 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 18174
I20260929 16:46:29.679313 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 18308
Sep 29 16:46:29 dist-test-slave-l31f krb5kdc[17934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.67@KRBTEST.COM
I20260929 16:46:29.696084 18450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:29.680171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48904 (local address 127.0.197.67:38181)
0929 16:46:29.680381 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:29.680389 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:29.680427 (+ 38us) server_negotiation.cc:407] Connection header received
0929 16:46:29.680785 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:29.680791 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:29.680902 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:29.680998 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:29.682589 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:29.683481 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:29.684733 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:29.685129 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:29.685235 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:29.689824 (+ 4589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:29.689841 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:29.689857 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:46:29.689904 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:29.693030 (+ 3126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:29.693867 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:29.693875 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:29.693877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:29.694036 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:29.694418 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:29.694421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:29.694423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:29.694796 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:29.695092 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:29.695506 (+ 414us) server_negotiation.cc:299] Negotiation successful
0929 16:46:29.695701 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260929 16:46:29.696784 18431 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:46:29.788703 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 18308
I20260929 16:46:29.806048 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 17947
I20260929 16:46:29.807276 18009 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:46:29.923065 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 17947
2026-09-29T16:46:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4495 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-29T16:46:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:46:29Z Disabled control of system clock
I20260929 16:46:30.007707 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:42433
--webserver_interface=127.0.197.126
--webserver_port=0
--builtin_ntp_servers=127.0.197.84:40309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.126:42433
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 16:46:30.361078 18490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:46:30.361414 18490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:46:30.361500 18490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:46:30.369036 18490 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:46:30.369189 18490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:46:30.369235 18490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:46:30.369292 18490 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:46:30.369369 18490 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:46:30.379297 18490 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:40309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.197.126:42433
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:42433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.197.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:46:30.380674 18490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:46:30.382598 18490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:46:30.395710 18496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:30.395691 18498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:30.395852 18495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:46:30.396875 18490 server_base.cc:1034] running on GCE node
I20260929 16:46:30.397792 18490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:46:30.399430 18490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:46:30.400655 18490 hybrid_clock.cc:648] HybridClock initialized: now 1790700390400616 us; error 40 us; skew 500 ppm
I20260929 16:46:30.401003 18490 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:46:30.408687 18490 webserver.cc:460] Webserver started at http://127.0.197.126:38805/ using document root <none> and password file <none>
I20260929 16:46:30.409502 18490 fs_manager.cc:359] Metadata directory not provided
I20260929 16:46:30.409610 18490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:46:30.409888 18490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:46:30.413195 18490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/master-0/data/instance:
uuid: "855457d956d64030ae26058c8ce2a8d8"
format_stamp: "Formatted at 2026-09-29 16:46:30 on dist-test-slave-l31f"
server_key: "4676c72beba4fb79b2c789ae3b061ca8"
server_key_iv: "d7f4a62bfb47166fd7d40437b45b2dc3"
server_key_version: "encryptionkey@0"
I20260929 16:46:30.414095 18490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/master-0/wal/instance:
uuid: "855457d956d64030ae26058c8ce2a8d8"
format_stamp: "Formatted at 2026-09-29 16:46:30 on dist-test-slave-l31f"
server_key: "4676c72beba4fb79b2c789ae3b061ca8"
server_key_iv: "d7f4a62bfb47166fd7d40437b45b2dc3"
server_key_version: "encryptionkey@0"
I20260929 16:46:30.420168 18490 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260929 16:46:30.424305 18504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:30.426014 18490 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 16:46:30.426272 18490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "855457d956d64030ae26058c8ce2a8d8"
format_stamp: "Formatted at 2026-09-29 16:46:30 on dist-test-slave-l31f"
server_key: "4676c72beba4fb79b2c789ae3b061ca8"
server_key_iv: "d7f4a62bfb47166fd7d40437b45b2dc3"
server_key_version: "encryptionkey@0"
I20260929 16:46:30.426546 18490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:46:30.445425 18490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:46:30.446388 18490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:46:30.446715 18490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:46:30.467443 18490 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.126:42433
I20260929 16:46:30.467463 18555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.126:42433 every 8 connection(s)
I20260929 16:46:30.468963 18490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:46:30.473246 18556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:46:30.474640 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 18490
I20260929 16:46:30.474841 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/master-0/wal/instance
I20260929 16:46:30.475353 789 external_mini_cluster.cc:1428] Setting key 6c5ced01c18ed15398eda384112c3682
I20260929 16:46:30.488345 18556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 855457d956d64030ae26058c8ce2a8d8: Bootstrap starting.
I20260929 16:46:30.490520 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:30.477552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37230 (local address 127.0.197.126:42433)
0929 16:46:30.478751 (+ 1199us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:30.478767 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:30.478818 (+ 51us) server_negotiation.cc:407] Connection header received
0929 16:46:30.480613 (+ 1795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:30.480643 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:30.481201 (+ 558us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:30.481581 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:30.482277 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:30.483672 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:30.485704 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:30.486463 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:30.486628 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:30.487984 (+ 1356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:30.487994 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:30.488016 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:46:30.488684 (+ 668us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:30.488774 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:46:30.488797 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:30.488889 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:30.489150 (+ 261us) server_negotiation.cc:299] Negotiation successful
0929 16:46:30.489527 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":466,"threads_started":1}
I20260929 16:46:30.492204 18556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 855457d956d64030ae26058c8ce2a8d8: Neither blocks nor log segments found. Creating new log.
I20260929 16:46:30.493350 18556 log.cc:824] T 00000000000000000000000000000000 P 855457d956d64030ae26058c8ce2a8d8: Log is configured to *not* fsync() on all Append() calls
I20260929 16:46:30.496508 18556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 855457d956d64030ae26058c8ce2a8d8: No bootstrap required, opened a new log
I20260929 16:46:30.504237 18556 raft_consensus.cc:348] T 00000000000000000000000000000000 P 855457d956d64030ae26058c8ce2a8d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "855457d956d64030ae26058c8ce2a8d8" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 42433 } }
I20260929 16:46:30.504678 18556 raft_consensus.cc:374] T 00000000000000000000000000000000 P 855457d956d64030ae26058c8ce2a8d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:46:30.504791 18556 raft_consensus.cc:729] T 00000000000000000000000000000000 P 855457d956d64030ae26058c8ce2a8d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 855457d956d64030ae26058c8ce2a8d8, State: Initialized, Role: FOLLOWER
I20260929 16:46:30.505476 18556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 855457d956d64030ae26058c8ce2a8d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "855457d956d64030ae26058c8ce2a8d8" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 42433 } }
I20260929 16:46:30.505731 18556 raft_consensus.cc:388] T 00000000000000000000000000000000 P 855457d956d64030ae26058c8ce2a8d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:46:30.505879 18556 raft_consensus.cc:482] T 00000000000000000000000000000000 P 855457d956d64030ae26058c8ce2a8d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:46:30.506057 18556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 855457d956d64030ae26058c8ce2a8d8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:46:30.508967 18556 raft_consensus.cc:504] T 00000000000000000000000000000000 P 855457d956d64030ae26058c8ce2a8d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "855457d956d64030ae26058c8ce2a8d8" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 42433 } }
I20260929 16:46:30.509588 18556 leader_election.cc:302] T 00000000000000000000000000000000 P 855457d956d64030ae26058c8ce2a8d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 855457d956d64030ae26058c8ce2a8d8; no voters:
I20260929 16:46:30.510339 18556 leader_election.cc:288] T 00000000000000000000000000000000 P 855457d956d64030ae26058c8ce2a8d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:46:30.510511 18561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 855457d956d64030ae26058c8ce2a8d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:46:30.511282 18561 raft_consensus.cc:686] T 00000000000000000000000000000000 P 855457d956d64030ae26058c8ce2a8d8 [term 1 LEADER]: Becoming Leader. State: Replica: 855457d956d64030ae26058c8ce2a8d8, State: Running, Role: LEADER
I20260929 16:46:30.511823 18561 consensus_queue.cc:237] T 00000000000000000000000000000000 P 855457d956d64030ae26058c8ce2a8d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "855457d956d64030ae26058c8ce2a8d8" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 42433 } }
I20260929 16:46:30.512632 18556 sys_catalog.cc:564] T 00000000000000000000000000000000 P 855457d956d64030ae26058c8ce2a8d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:46:30.514847 18563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 855457d956d64030ae26058c8ce2a8d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 855457d956d64030ae26058c8ce2a8d8. Latest consensus state: current_term: 1 leader_uuid: "855457d956d64030ae26058c8ce2a8d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "855457d956d64030ae26058c8ce2a8d8" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 42433 } } }
I20260929 16:46:30.515156 18562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 855457d956d64030ae26058c8ce2a8d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "855457d956d64030ae26058c8ce2a8d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "855457d956d64030ae26058c8ce2a8d8" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 42433 } } }
I20260929 16:46:30.515457 18563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 855457d956d64030ae26058c8ce2a8d8 [sys.catalog]: This master's current role is: LEADER
I20260929 16:46:30.515457 18562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 855457d956d64030ae26058c8ce2a8d8 [sys.catalog]: This master's current role is: LEADER
I20260929 16:46:30.517247 18569 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:46:30.523495 18569 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:46:30.533826 18569 catalog_manager.cc:1347] Generated new cluster ID: 3a2622a7bde44dc4a1b3c64cad3297ed
I20260929 16:46:30.533982 18569 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:46:30.582046 18569 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:46:30.583721 18569 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:46:30.594565 18569 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 855457d956d64030ae26058c8ce2a8d8: Generated new TSK 0
I20260929 16:46:30.595638 18569 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:46:30.605216 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.65:0
--local_ip_for_outbound_sockets=127.0.197.65
--webserver_interface=127.0.197.65
--webserver_port=0
--tserver_master_addrs=127.0.197.126:42433
--builtin_ntp_servers=127.0.197.84:40309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 16:46:30.966591 18580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:46:30.966925 18580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:46:30.967237 18580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:46:30.975400 18580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:46:30.975526 18580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:46:30.975701 18580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.65
I20260929 16:46:30.985751 18580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:40309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.197.65
--webserver_port=0
--tserver_master_addrs=127.0.197.126:42433
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.65
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:46:30.987223 18580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:46:30.989151 18580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:46:31.007392 18586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:31.007737 18588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:31.008293 18585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:46:31.008805 18580 server_base.cc:1034] running on GCE node
I20260929 16:46:31.009543 18580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:46:31.010752 18580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:46:31.012009 18580 hybrid_clock.cc:648] HybridClock initialized: now 1790700391011917 us; error 86 us; skew 500 ppm
I20260929 16:46:31.012399 18580 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:46:31.015236 18580 webserver.cc:460] Webserver started at http://127.0.197.65:43489/ using document root <none> and password file <none>
I20260929 16:46:31.016013 18580 fs_manager.cc:359] Metadata directory not provided
I20260929 16:46:31.016161 18580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:46:31.016503 18580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:46:31.019927 18580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/ts-0/data/instance:
uuid: "c7f9427b9d964322bd42454d632b5394"
format_stamp: "Formatted at 2026-09-29 16:46:31 on dist-test-slave-l31f"
server_key: "73cc420a2dbc84d3c6f5d3a13f49914a"
server_key_iv: "5fae7003d27b14e7293276c6c4028e6e"
server_key_version: "encryptionkey@0"
I20260929 16:46:31.020838 18580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/ts-0/wal/instance:
uuid: "c7f9427b9d964322bd42454d632b5394"
format_stamp: "Formatted at 2026-09-29 16:46:31 on dist-test-slave-l31f"
server_key: "73cc420a2dbc84d3c6f5d3a13f49914a"
server_key_iv: "5fae7003d27b14e7293276c6c4028e6e"
server_key_version: "encryptionkey@0"
I20260929 16:46:31.026681 18580 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 16:46:31.030443 18594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:31.032051 18580 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:46:31.032359 18580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/ts-0/wal
uuid: "c7f9427b9d964322bd42454d632b5394"
format_stamp: "Formatted at 2026-09-29 16:46:31 on dist-test-slave-l31f"
server_key: "73cc420a2dbc84d3c6f5d3a13f49914a"
server_key_iv: "5fae7003d27b14e7293276c6c4028e6e"
server_key_version: "encryptionkey@0"
I20260929 16:46:31.032630 18580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:46:31.061118 18580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:46:31.062062 18580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:46:31.062397 18580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:46:31.063482 18580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:46:31.065161 18580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:46:31.065281 18580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:31.065423 18580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:46:31.065511 18580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:31.101436 18580 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.65:45283
I20260929 16:46:31.101500 18706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.65:45283 every 8 connection(s)
I20260929 16:46:31.102866 18580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
I20260929 16:46:31.106194 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 18580
I20260929 16:46:31.106398 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/ts-0/wal/instance
I20260929 16:46:31.107040 789 external_mini_cluster.cc:1428] Setting key 59e668200796aef9ecdff98b1563bb60
I20260929 16:46:31.110268 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.66:0
--local_ip_for_outbound_sockets=127.0.197.66
--webserver_interface=127.0.197.66
--webserver_port=0
--tserver_master_addrs=127.0.197.126:42433
--builtin_ntp_servers=127.0.197.84:40309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 16:46:31.119081 18709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:31.105791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:55115 (local address 127.0.197.126:42433)
0929 16:46:31.106372 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:31.106381 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:31.108143 (+ 1762us) server_negotiation.cc:407] Connection header received
0929 16:46:31.110556 (+ 2413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:31.110565 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:31.110663 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:31.110790 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:31.112127 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:31.113319 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:31.115509 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:31.116256 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:31.116429 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:31.118124 (+ 1695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:31.118129 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:31.118132 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:46:31.118153 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:31.118195 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:46:31.118208 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:31.118276 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:31.118625 (+ 349us) server_negotiation.cc:299] Negotiation successful
0929 16:46:31.118755 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":322,"threads_started":1}
I20260929 16:46:31.120766 18710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:31.106444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:42433 (local address 127.0.197.65:55115)
0929 16:46:31.107672 (+ 1228us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:31.107736 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 16:46:31.110166 (+ 2430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:46:31.111158 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:46:31.111167 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:46:31.111270 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:46:31.111904 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:31.111935 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:31.113557 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:31.113562 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:31.115284 (+ 1722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:31.115303 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:31.116529 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:31.116535 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:31.116784 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:31.117710 (+ 926us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:46:31.117712 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:46:31.117767 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:46:31.117867 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:46:31.118295 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:46:31.118311 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:46:31.118323 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 16:46:31.118555 (+ 232us) client_negotiation.cc:240] Negotiation successful
0929 16:46:31.119115 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":421,"threads_started":1}
I20260929 16:46:31.122767 18707 heartbeater.cc:347] Connected to a master server at 127.0.197.126:42433
I20260929 16:46:31.123230 18707 heartbeater.cc:464] Registering TS with master...
I20260929 16:46:31.124066 18707 heartbeater.cc:511] Master 127.0.197.126:42433 requested a full tablet report, sending...
I20260929 16:46:31.126755 18521 ts_manager.cc:194] Registered new tserver with Master: c7f9427b9d964322bd42454d632b5394 (127.0.197.65:45283)
I20260929 16:46:31.129423 18521 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.197.65:55115
W20260929 16:46:31.464254 18711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:46:31.464576 18711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:46:31.464668 18711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:46:31.473052 18711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:46:31.473225 18711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:46:31.473408 18711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.66
I20260929 16:46:31.484050 18711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:40309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.197.66
--webserver_port=0
--tserver_master_addrs=127.0.197.126:42433
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.66
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:46:31.485437 18711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:46:31.487460 18711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:46:31.501482 18716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:31.501552 18719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:31.501511 18717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:46:31.502396 18711 server_base.cc:1034] running on GCE node
I20260929 16:46:31.503145 18711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:46:31.504192 18711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:46:31.505412 18711 hybrid_clock.cc:648] HybridClock initialized: now 1790700391505345 us; error 56 us; skew 500 ppm
I20260929 16:46:31.505776 18711 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:46:31.508180 18711 webserver.cc:460] Webserver started at http://127.0.197.66:39181/ using document root <none> and password file <none>
I20260929 16:46:31.508910 18711 fs_manager.cc:359] Metadata directory not provided
I20260929 16:46:31.509001 18711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:46:31.509270 18711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:46:31.512552 18711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/ts-1/data/instance:
uuid: "da3610a36c244fd2a23daf995f4ad3b0"
format_stamp: "Formatted at 2026-09-29 16:46:31 on dist-test-slave-l31f"
server_key: "dbf6ecac5dcd9b67a43a0e25956bd958"
server_key_iv: "421c2051249f951781c76acd57b8b9b9"
server_key_version: "encryptionkey@0"
I20260929 16:46:31.513412 18711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/ts-1/wal/instance:
uuid: "da3610a36c244fd2a23daf995f4ad3b0"
format_stamp: "Formatted at 2026-09-29 16:46:31 on dist-test-slave-l31f"
server_key: "dbf6ecac5dcd9b67a43a0e25956bd958"
server_key_iv: "421c2051249f951781c76acd57b8b9b9"
server_key_version: "encryptionkey@0"
I20260929 16:46:31.519444 18711 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 16:46:31.523417 18725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:31.524937 18711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:46:31.525143 18711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/ts-1/wal
uuid: "da3610a36c244fd2a23daf995f4ad3b0"
format_stamp: "Formatted at 2026-09-29 16:46:31 on dist-test-slave-l31f"
server_key: "dbf6ecac5dcd9b67a43a0e25956bd958"
server_key_iv: "421c2051249f951781c76acd57b8b9b9"
server_key_version: "encryptionkey@0"
I20260929 16:46:31.525390 18711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:46:31.544480 18711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:46:31.545408 18711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:46:31.545746 18711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:46:31.546839 18711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:46:31.548480 18711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:46:31.548579 18711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:31.548738 18711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:46:31.548830 18711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:31.585870 18711 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.66:45429
I20260929 16:46:31.585922 18837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.66:45429 every 8 connection(s)
I20260929 16:46:31.587363 18711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
I20260929 16:46:31.592407 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 18711
I20260929 16:46:31.592615 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/ts-1/wal/instance
I20260929 16:46:31.593050 789 external_mini_cluster.cc:1428] Setting key f1dcc68677e7b14d8e10240fbf41f372
I20260929 16:46:31.596100 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.67:0
--local_ip_for_outbound_sockets=127.0.197.67
--webserver_interface=127.0.197.67
--webserver_port=0
--tserver_master_addrs=127.0.197.126:42433
--builtin_ntp_servers=127.0.197.84:40309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 16:46:31.601627 18709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:31.590440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:39381 (local address 127.0.197.126:42433)
0929 16:46:31.590681 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:31.590688 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:31.592199 (+ 1511us) server_negotiation.cc:407] Connection header received
0929 16:46:31.594383 (+ 2184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:31.594389 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:31.594486 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:31.594609 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:31.595983 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:31.596783 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:31.598514 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:31.598915 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:31.599088 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:31.600734 (+ 1646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:31.600740 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:31.600743 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:46:31.600765 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:31.600800 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:46:31.600812 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:31.600919 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:31.601171 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 16:46:31.601289 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260929 16:46:31.602753 18840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:31.590988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:42433 (local address 127.0.197.66:39381)
0929 16:46:31.592021 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:31.592064 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 16:46:31.594078 (+ 2014us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:46:31.594925 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:46:31.594938 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:46:31.595125 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:46:31.595784 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:31.595804 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:31.596956 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:31.596968 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:31.598309 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:31.598324 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:31.599209 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:31.599219 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:31.599510 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:31.600355 (+ 845us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:46:31.600357 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:46:31.600402 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:46:31.600507 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:46:31.600919 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:46:31.600932 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:46:31.600942 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 16:46:31.601143 (+ 201us) client_negotiation.cc:240] Negotiation successful
0929 16:46:31.601614 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":329,"threads_started":1}
I20260929 16:46:31.604465 18838 heartbeater.cc:347] Connected to a master server at 127.0.197.126:42433
I20260929 16:46:31.604802 18838 heartbeater.cc:464] Registering TS with master...
I20260929 16:46:31.605695 18838 heartbeater.cc:511] Master 127.0.197.126:42433 requested a full tablet report, sending...
I20260929 16:46:31.607326 18521 ts_manager.cc:194] Registered new tserver with Master: da3610a36c244fd2a23daf995f4ad3b0 (127.0.197.66:45429)
I20260929 16:46:31.608168 18521 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.197.66:39381
W20260929 16:46:31.954502 18841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:46:31.955008 18841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:46:31.955199 18841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:46:31.963521 18841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:46:31.963678 18841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:46:31.963856 18841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.67
I20260929 16:46:31.973932 18841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:40309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.197.67
--webserver_port=0
--tserver_master_addrs=127.0.197.126:42433
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.67
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:46:31.975410 18841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:46:31.977286 18841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:46:31.991799 18846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:31.991837 18847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:31.991882 18849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:46:31.992847 18841 server_base.cc:1034] running on GCE node
I20260929 16:46:31.993537 18841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:46:31.994673 18841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:46:31.995882 18841 hybrid_clock.cc:648] HybridClock initialized: now 1790700391995830 us; error 53 us; skew 500 ppm
I20260929 16:46:31.996224 18841 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:46:31.998857 18841 webserver.cc:460] Webserver started at http://127.0.197.67:35575/ using document root <none> and password file <none>
I20260929 16:46:31.999754 18841 fs_manager.cc:359] Metadata directory not provided
I20260929 16:46:31.999868 18841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:46:32.000291 18841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:46:32.003772 18841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/ts-2/data/instance:
uuid: "b3e24b2b75be48a6b602b3085da5dba3"
format_stamp: "Formatted at 2026-09-29 16:46:31 on dist-test-slave-l31f"
server_key: "84b6edaddbcbcaf277d08d51b71e5026"
server_key_iv: "bf53b0281da59c63f33c4fbd9d07cc5d"
server_key_version: "encryptionkey@0"
I20260929 16:46:32.004680 18841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/ts-2/wal/instance:
uuid: "b3e24b2b75be48a6b602b3085da5dba3"
format_stamp: "Formatted at 2026-09-29 16:46:31 on dist-test-slave-l31f"
server_key: "84b6edaddbcbcaf277d08d51b71e5026"
server_key_iv: "bf53b0281da59c63f33c4fbd9d07cc5d"
server_key_version: "encryptionkey@0"
I20260929 16:46:32.010449 18841 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 16:46:32.015130 18855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:32.016748 18841 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:46:32.016948 18841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/ts-2/wal
uuid: "b3e24b2b75be48a6b602b3085da5dba3"
format_stamp: "Formatted at 2026-09-29 16:46:31 on dist-test-slave-l31f"
server_key: "84b6edaddbcbcaf277d08d51b71e5026"
server_key_iv: "bf53b0281da59c63f33c4fbd9d07cc5d"
server_key_version: "encryptionkey@0"
I20260929 16:46:32.017201 18841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:46:32.036073 18841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:46:32.037045 18841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:46:32.037443 18841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:46:32.038514 18841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:46:32.040292 18841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:46:32.040416 18841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:32.040594 18841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:46:32.040695 18841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:32.076956 18841 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.67:46627
I20260929 16:46:32.077016 18967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.67:46627 every 8 connection(s)
I20260929 16:46:32.078387 18841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
I20260929 16:46:32.083734 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 18841
I20260929 16:46:32.083914 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/ts-2/wal/instance
I20260929 16:46:32.084368 789 external_mini_cluster.cc:1428] Setting key ae9cc787f1e1e0d85dfaa77b9d347a0c
I20260929 16:46:32.091882 18709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:32.081312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:42315 (local address 127.0.197.126:42433)
0929 16:46:32.081506 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:32.081510 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:32.082942 (+ 1432us) server_negotiation.cc:407] Connection header received
0929 16:46:32.085089 (+ 2147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:32.085095 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:32.085205 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:32.085321 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:32.086764 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:32.087710 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:32.089284 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:32.089702 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:32.089798 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:32.091211 (+ 1413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:32.091216 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:32.091219 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:46:32.091237 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:32.091267 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:46:32.091278 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:32.091347 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:32.091547 (+ 200us) server_negotiation.cc:299] Negotiation successful
0929 16:46:32.091629 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 16:46:32.092995 18970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:32.081844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:42433 (local address 127.0.197.67:42315)
0929 16:46:32.082747 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:32.082783 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 16:46:32.084719 (+ 1936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:46:32.085658 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:46:32.085671 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:46:32.085793 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:46:32.086580 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:32.086608 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:32.087862 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:32.087866 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:32.089141 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:32.089154 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:32.089894 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:32.089899 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:32.090102 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:32.090856 (+ 754us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:46:32.090860 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:46:32.090902 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:46:32.090986 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:46:32.091361 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:46:32.091369 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:46:32.091375 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 16:46:32.091523 (+ 148us) client_negotiation.cc:240] Negotiation successful
0929 16:46:32.091878 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":280,"threads_started":1}
I20260929 16:46:32.094527 18968 heartbeater.cc:347] Connected to a master server at 127.0.197.126:42433
I20260929 16:46:32.094856 18968 heartbeater.cc:464] Registering TS with master...
I20260929 16:46:32.095726 18968 heartbeater.cc:511] Master 127.0.197.126:42433 requested a full tablet report, sending...
I20260929 16:46:32.097404 18521 ts_manager.cc:194] Registered new tserver with Master: b3e24b2b75be48a6b602b3085da5dba3 (127.0.197.67:46627)
I20260929 16:46:32.098394 18521 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.197.67:42315
I20260929 16:46:32.100011 789 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:46:32.111785 18709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:32.105380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37232 (local address 127.0.197.126:42433)
0929 16:46:32.105603 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:32.105609 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:32.105983 (+ 374us) server_negotiation.cc:407] Connection header received
0929 16:46:32.106142 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:32.106148 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:32.106232 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:32.106355 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:32.106884 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:32.107922 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:32.109469 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:32.109882 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:32.109978 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:32.111162 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:32.111167 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:32.111170 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:46:32.111190 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:32.111221 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:46:32.111234 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:32.111312 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:32.111444 (+ 132us) server_negotiation.cc:299] Negotiation successful
0929 16:46:32.111527 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 16:46:32.116123 18521 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37232:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:46:32.123589 18521 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:46:32.133607 18707 heartbeater.cc:503] Master 127.0.197.126:42433 was elected leader, sending a full tablet report...
I20260929 16:46:32.170390 18984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:32.138920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57610 (local address 127.0.197.66:45429)
0929 16:46:32.154321 (+ 15401us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:32.154336 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:32.162548 (+ 8212us) server_negotiation.cc:407] Connection header received
0929 16:46:32.162718 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:32.162725 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:32.163054 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:32.163257 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:46:32.163861 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:32.165279 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:32.167421 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:32.168954 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:32.169132 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:32.169422 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:32.169742 (+ 320us) server_negotiation.cc:299] Negotiation successful
0929 16:46:32.169958 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15112,"thread_start_us":15407,"threads_started":1}
I20260929 16:46:32.170553 18981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:32.138743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.66:45429 (local address 127.0.0.1:57610)
0929 16:46:32.162400 (+ 23657us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:32.162415 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:46:32.162569 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:46:32.163275 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:46:32.163280 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:46:32.163307 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:46:32.163660 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:32.163672 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:32.165430 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:32.165435 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:32.167187 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:32.167206 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:32.169185 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:32.169193 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:32.169455 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:32.169520 (+ 65us) client_negotiation.cc:769] Sending connection context
0929 16:46:32.169694 (+ 174us) client_negotiation.cc:240] Negotiation successful
0929 16:46:32.169958 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23539,"mutex_wait_us":1,"thread_start_us":21053,"threads_started":1}
I20260929 16:46:32.171734 18982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:32.138315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54882 (local address 127.0.197.67:46627)
0929 16:46:32.162104 (+ 23789us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:32.162112 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:32.162148 (+ 36us) server_negotiation.cc:407] Connection header received
0929 16:46:32.162252 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:32.162259 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:32.162533 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:32.162731 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:46:32.163398 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:32.164925 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:32.167817 (+ 2892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:32.169203 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:32.169339 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:32.169616 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:32.170781 (+ 1165us) server_negotiation.cc:299] Negotiation successful
0929 16:46:32.170997 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23572,"thread_start_us":23398,"threads_started":1}
I20260929 16:46:32.172004 18985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:32.138962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56442 (local address 127.0.197.65:45283)
0929 16:46:32.160586 (+ 21624us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:32.160597 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:32.160634 (+ 37us) server_negotiation.cc:407] Connection header received
0929 16:46:32.160747 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:32.160757 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:32.161075 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:32.161306 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:46:32.162157 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:32.163978 (+ 1821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:32.166825 (+ 2847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:32.168383 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:32.168525 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:32.170126 (+ 1601us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:32.171328 (+ 1202us) server_negotiation.cc:299] Negotiation successful
0929 16:46:32.171630 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21397,"thread_start_us":21203,"threads_started":1}
I20260929 16:46:32.172248 18980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:32.137858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.67:46627 (local address 127.0.0.1:54882)
0929 16:46:32.159549 (+ 21691us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:32.159630 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0929 16:46:32.159846 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:46:32.162726 (+ 2880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:46:32.162730 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:46:32.162753 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:46:32.163184 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:32.163201 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:32.165094 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:32.165101 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:32.167585 (+ 2484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:32.167601 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:32.169731 (+ 2130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:32.169736 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:32.169912 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:32.169962 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 16:46:32.170040 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 16:46:32.170413 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21276,"mutex_wait_us":77,"spinlock_wait_cycles":13312,"thread_start_us":21030,"threads_started":1}
I20260929 16:46:32.172371 18983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:32.138161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:45283 (local address 127.0.0.1:56442)
0929 16:46:32.159552 (+ 21391us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:32.159616 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 16:46:32.159846 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:46:32.161315 (+ 1469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:46:32.161324 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:46:32.161358 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:46:32.161873 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:32.161891 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:32.164117 (+ 2226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:32.164126 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:32.166570 (+ 2444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:32.166588 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:32.169674 (+ 3086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:32.169682 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:32.169912 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:32.171129 (+ 1217us) client_negotiation.cc:769] Sending connection context
0929 16:46:32.171238 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 16:46:32.171606 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21241,"mutex_wait_us":1198,"spinlock_wait_cycles":8576,"thread_start_us":23979,"threads_started":1}
I20260929 16:46:32.175215 18773 tablet_service.cc:1467] Processing CreateTablet for tablet d4b22124b4f24384b3a15373b6ade7cd (DEFAULT_TABLE table=test-table [id=7a8164dd8aff4103a32f5353e767654d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:46:32.176048 18903 tablet_service.cc:1467] Processing CreateTablet for tablet d4b22124b4f24384b3a15373b6ade7cd (DEFAULT_TABLE table=test-table [id=7a8164dd8aff4103a32f5353e767654d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:46:32.176837 18773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4b22124b4f24384b3a15373b6ade7cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:46:32.177501 18903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4b22124b4f24384b3a15373b6ade7cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:46:32.177209 18642 tablet_service.cc:1467] Processing CreateTablet for tablet d4b22124b4f24384b3a15373b6ade7cd (DEFAULT_TABLE table=test-table [id=7a8164dd8aff4103a32f5353e767654d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:46:32.178606 18642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4b22124b4f24384b3a15373b6ade7cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:46:32.191315 18986 tablet_bootstrap.cc:492] T d4b22124b4f24384b3a15373b6ade7cd P c7f9427b9d964322bd42454d632b5394: Bootstrap starting.
I20260929 16:46:32.193627 18988 tablet_bootstrap.cc:492] T d4b22124b4f24384b3a15373b6ade7cd P da3610a36c244fd2a23daf995f4ad3b0: Bootstrap starting.
I20260929 16:46:32.195699 18986 tablet_bootstrap.cc:654] T d4b22124b4f24384b3a15373b6ade7cd P c7f9427b9d964322bd42454d632b5394: Neither blocks nor log segments found. Creating new log.
I20260929 16:46:32.197136 18986 log.cc:824] T d4b22124b4f24384b3a15373b6ade7cd P c7f9427b9d964322bd42454d632b5394: Log is configured to *not* fsync() on all Append() calls
I20260929 16:46:32.197582 18988 tablet_bootstrap.cc:654] T d4b22124b4f24384b3a15373b6ade7cd P da3610a36c244fd2a23daf995f4ad3b0: Neither blocks nor log segments found. Creating new log.
I20260929 16:46:32.199613 18988 log.cc:824] T d4b22124b4f24384b3a15373b6ade7cd P da3610a36c244fd2a23daf995f4ad3b0: Log is configured to *not* fsync() on all Append() calls
I20260929 16:46:32.200357 18987 tablet_bootstrap.cc:492] T d4b22124b4f24384b3a15373b6ade7cd P b3e24b2b75be48a6b602b3085da5dba3: Bootstrap starting.
I20260929 16:46:32.203517 18988 tablet_bootstrap.cc:492] T d4b22124b4f24384b3a15373b6ade7cd P da3610a36c244fd2a23daf995f4ad3b0: No bootstrap required, opened a new log
I20260929 16:46:32.203856 18988 ts_tablet_manager.cc:1397] T d4b22124b4f24384b3a15373b6ade7cd P da3610a36c244fd2a23daf995f4ad3b0: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260929 16:46:32.206799 18987 tablet_bootstrap.cc:654] T d4b22124b4f24384b3a15373b6ade7cd P b3e24b2b75be48a6b602b3085da5dba3: Neither blocks nor log segments found. Creating new log.
I20260929 16:46:32.207762 18986 tablet_bootstrap.cc:492] T d4b22124b4f24384b3a15373b6ade7cd P c7f9427b9d964322bd42454d632b5394: No bootstrap required, opened a new log
I20260929 16:46:32.208060 18986 ts_tablet_manager.cc:1397] T d4b22124b4f24384b3a15373b6ade7cd P c7f9427b9d964322bd42454d632b5394: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.003s
I20260929 16:46:32.208170 18987 log.cc:824] T d4b22124b4f24384b3a15373b6ade7cd P b3e24b2b75be48a6b602b3085da5dba3: Log is configured to *not* fsync() on all Append() calls
I20260929 16:46:32.211750 18987 tablet_bootstrap.cc:492] T d4b22124b4f24384b3a15373b6ade7cd P b3e24b2b75be48a6b602b3085da5dba3: No bootstrap required, opened a new log
I20260929 16:46:32.212059 18987 ts_tablet_manager.cc:1397] T d4b22124b4f24384b3a15373b6ade7cd P b3e24b2b75be48a6b602b3085da5dba3: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260929 16:46:32.214805 18988 raft_consensus.cc:348] T d4b22124b4f24384b3a15373b6ade7cd P da3610a36c244fd2a23daf995f4ad3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e24b2b75be48a6b602b3085da5dba3" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46627 } } peers { permanent_uuid: "c7f9427b9d964322bd42454d632b5394" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 45283 } } peers { permanent_uuid: "da3610a36c244fd2a23daf995f4ad3b0" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 45429 } }
I20260929 16:46:32.215389 18988 raft_consensus.cc:374] T d4b22124b4f24384b3a15373b6ade7cd P da3610a36c244fd2a23daf995f4ad3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:46:32.215492 18988 raft_consensus.cc:729] T d4b22124b4f24384b3a15373b6ade7cd P da3610a36c244fd2a23daf995f4ad3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3610a36c244fd2a23daf995f4ad3b0, State: Initialized, Role: FOLLOWER
I20260929 16:46:32.216230 18988 consensus_queue.cc:260] T d4b22124b4f24384b3a15373b6ade7cd P da3610a36c244fd2a23daf995f4ad3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e24b2b75be48a6b602b3085da5dba3" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46627 } } peers { permanent_uuid: "c7f9427b9d964322bd42454d632b5394" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 45283 } } peers { permanent_uuid: "da3610a36c244fd2a23daf995f4ad3b0" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 45429 } }
I20260929 16:46:32.216629 18986 raft_consensus.cc:348] T d4b22124b4f24384b3a15373b6ade7cd P c7f9427b9d964322bd42454d632b5394 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e24b2b75be48a6b602b3085da5dba3" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46627 } } peers { permanent_uuid: "c7f9427b9d964322bd42454d632b5394" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 45283 } } peers { permanent_uuid: "da3610a36c244fd2a23daf995f4ad3b0" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 45429 } }
I20260929 16:46:32.217212 18986 raft_consensus.cc:374] T d4b22124b4f24384b3a15373b6ade7cd P c7f9427b9d964322bd42454d632b5394 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:46:32.217314 18986 raft_consensus.cc:729] T d4b22124b4f24384b3a15373b6ade7cd P c7f9427b9d964322bd42454d632b5394 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7f9427b9d964322bd42454d632b5394, State: Initialized, Role: FOLLOWER
I20260929 16:46:32.217710 18838 heartbeater.cc:503] Master 127.0.197.126:42433 was elected leader, sending a full tablet report...
I20260929 16:46:32.217993 18986 consensus_queue.cc:260] T d4b22124b4f24384b3a15373b6ade7cd P c7f9427b9d964322bd42454d632b5394 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e24b2b75be48a6b602b3085da5dba3" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46627 } } peers { permanent_uuid: "c7f9427b9d964322bd42454d632b5394" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 45283 } } peers { permanent_uuid: "da3610a36c244fd2a23daf995f4ad3b0" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 45429 } }
I20260929 16:46:32.218235 18988 ts_tablet_manager.cc:1428] T d4b22124b4f24384b3a15373b6ade7cd P da3610a36c244fd2a23daf995f4ad3b0: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260929 16:46:32.223723 18986 ts_tablet_manager.cc:1428] T d4b22124b4f24384b3a15373b6ade7cd P c7f9427b9d964322bd42454d632b5394: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260929 16:46:32.223970 18987 raft_consensus.cc:348] T d4b22124b4f24384b3a15373b6ade7cd P b3e24b2b75be48a6b602b3085da5dba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e24b2b75be48a6b602b3085da5dba3" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46627 } } peers { permanent_uuid: "c7f9427b9d964322bd42454d632b5394" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 45283 } } peers { permanent_uuid: "da3610a36c244fd2a23daf995f4ad3b0" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 45429 } }
I20260929 16:46:32.224515 18987 raft_consensus.cc:374] T d4b22124b4f24384b3a15373b6ade7cd P b3e24b2b75be48a6b602b3085da5dba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:46:32.224633 18987 raft_consensus.cc:729] T d4b22124b4f24384b3a15373b6ade7cd P b3e24b2b75be48a6b602b3085da5dba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3e24b2b75be48a6b602b3085da5dba3, State: Initialized, Role: FOLLOWER
I20260929 16:46:32.225332 18987 consensus_queue.cc:260] T d4b22124b4f24384b3a15373b6ade7cd P b3e24b2b75be48a6b602b3085da5dba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e24b2b75be48a6b602b3085da5dba3" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46627 } } peers { permanent_uuid: "c7f9427b9d964322bd42454d632b5394" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 45283 } } peers { permanent_uuid: "da3610a36c244fd2a23daf995f4ad3b0" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 45429 } }
I20260929 16:46:32.226629 18968 heartbeater.cc:503] Master 127.0.197.126:42433 was elected leader, sending a full tablet report...
I20260929 16:46:32.227423 18987 ts_tablet_manager.cc:1428] T d4b22124b4f24384b3a15373b6ade7cd P b3e24b2b75be48a6b602b3085da5dba3: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
W20260929 16:46:32.330449 18969 tablet.cc:2369] T d4b22124b4f24384b3a15373b6ade7cd P b3e24b2b75be48a6b602b3085da5dba3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:46:32.339758 18839 tablet.cc:2369] T d4b22124b4f24384b3a15373b6ade7cd P da3610a36c244fd2a23daf995f4ad3b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:46:32.356072 18708 tablet.cc:2369] T d4b22124b4f24384b3a15373b6ade7cd P c7f9427b9d964322bd42454d632b5394: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:46:32.643736 18994 raft_consensus.cc:482] T d4b22124b4f24384b3a15373b6ade7cd P b3e24b2b75be48a6b602b3085da5dba3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:46:32.644032 18994 raft_consensus.cc:504] T d4b22124b4f24384b3a15373b6ade7cd P b3e24b2b75be48a6b602b3085da5dba3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e24b2b75be48a6b602b3085da5dba3" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46627 } } peers { permanent_uuid: "c7f9427b9d964322bd42454d632b5394" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 45283 } } peers { permanent_uuid: "da3610a36c244fd2a23daf995f4ad3b0" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 45429 } }
I20260929 16:46:32.645699 18994 leader_election.cc:288] T d4b22124b4f24384b3a15373b6ade7cd P b3e24b2b75be48a6b602b3085da5dba3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7f9427b9d964322bd42454d632b5394 (127.0.197.65:45283), da3610a36c244fd2a23daf995f4ad3b0 (127.0.197.66:45429)
I20260929 16:46:32.652897 18985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:32.646141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:33949 (local address 127.0.197.65:45283)
0929 16:46:32.646404 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:32.646414 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:32.647214 (+ 800us) server_negotiation.cc:407] Connection header received
0929 16:46:32.647402 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:32.647406 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:32.647508 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:32.647618 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:46:32.648279 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:32.649015 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:32.650959 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:32.651823 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:32.652016 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:32.652089 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:32.652453 (+ 364us) server_negotiation.cc:299] Negotiation successful
0929 16:46:32.652610 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 16:46:32.652902 18984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:32.646298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:34813 (local address 127.0.197.66:45429)
0929 16:46:32.646558 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:32.646566 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:32.647058 (+ 492us) server_negotiation.cc:407] Connection header received
0929 16:46:32.647330 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:32.647339 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:32.647433 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:32.647510 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:46:32.648187 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:32.648905 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:32.650931 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:32.651814 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:32.652011 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:32.652089 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:32.652466 (+ 377us) server_negotiation.cc:299] Negotiation successful
0929 16:46:32.652616 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 16:46:32.652981 18996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:32.646054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:45283 (local address 127.0.197.67:33949)
0929 16:46:32.647087 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:32.647106 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:46:32.647253 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:46:32.647651 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:46:32.647659 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:46:32.647705 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:46:32.648069 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:32.648079 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:32.649180 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:32.649184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:32.650706 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:32.650719 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:32.652000 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:32.652006 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:32.652240 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:32.652299 (+ 59us) client_negotiation.cc:769] Sending connection context
0929 16:46:32.652428 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 16:46:32.652624 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":841,"mutex_wait_us":195,"thread_start_us":534,"threads_started":1}
I20260929 16:46:32.653025 18995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:32.646045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.66:45429 (local address 127.0.197.67:34813)
0929 16:46:32.646882 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:32.646907 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 16:46:32.647143 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:46:32.647558 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:46:32.647565 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:46:32.647596 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:46:32.648014 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:32.648024 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:32.649053 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:32.649061 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:32.650687 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:32.650702 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:32.652032 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:32.652038 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:32.652252 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:32.652305 (+ 53us) client_negotiation.cc:769] Sending connection context
0929 16:46:32.652416 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 16:46:32.652621 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"mutex_wait_us":103,"spinlock_wait_cycles":27520,"thread_start_us":511,"threads_started":1}
I20260929 16:46:32.654222 18793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4b22124b4f24384b3a15373b6ade7cd" candidate_uuid: "b3e24b2b75be48a6b602b3085da5dba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3610a36c244fd2a23daf995f4ad3b0" is_pre_election: true
I20260929 16:46:32.654225 18662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4b22124b4f24384b3a15373b6ade7cd" candidate_uuid: "b3e24b2b75be48a6b602b3085da5dba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7f9427b9d964322bd42454d632b5394" is_pre_election: true
I20260929 16:46:32.654865 18662 raft_consensus.cc:2454] T d4b22124b4f24384b3a15373b6ade7cd P c7f9427b9d964322bd42454d632b5394 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3e24b2b75be48a6b602b3085da5dba3 in term 0.
I20260929 16:46:32.654865 18793 raft_consensus.cc:2454] T d4b22124b4f24384b3a15373b6ade7cd P da3610a36c244fd2a23daf995f4ad3b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3e24b2b75be48a6b602b3085da5dba3 in term 0.
I20260929 16:46:32.655695 18858 leader_election.cc:302] T d4b22124b4f24384b3a15373b6ade7cd P b3e24b2b75be48a6b602b3085da5dba3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: b3e24b2b75be48a6b602b3085da5dba3, c7f9427b9d964322bd42454d632b5394, da3610a36c244fd2a23daf995f4ad3b0; no voters:
I20260929 16:46:32.656186 18994 raft_consensus.cc:2781] T d4b22124b4f24384b3a15373b6ade7cd P b3e24b2b75be48a6b602b3085da5dba3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:46:32.656358 18994 raft_consensus.cc:482] T d4b22124b4f24384b3a15373b6ade7cd P b3e24b2b75be48a6b602b3085da5dba3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:46:32.656466 18994 raft_consensus.cc:3037] T d4b22124b4f24384b3a15373b6ade7cd P b3e24b2b75be48a6b602b3085da5dba3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:46:32.658720 18994 raft_consensus.cc:504] T d4b22124b4f24384b3a15373b6ade7cd P b3e24b2b75be48a6b602b3085da5dba3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e24b2b75be48a6b602b3085da5dba3" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46627 } } peers { permanent_uuid: "c7f9427b9d964322bd42454d632b5394" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 45283 } } peers { permanent_uuid: "da3610a36c244fd2a23daf995f4ad3b0" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 45429 } }
I20260929 16:46:32.659476 18994 leader_election.cc:288] T d4b22124b4f24384b3a15373b6ade7cd P b3e24b2b75be48a6b602b3085da5dba3 [CANDIDATE]: Term 1 election: Requested vote from peers c7f9427b9d964322bd42454d632b5394 (127.0.197.65:45283), da3610a36c244fd2a23daf995f4ad3b0 (127.0.197.66:45429)
I20260929 16:46:32.660151 18793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4b22124b4f24384b3a15373b6ade7cd" candidate_uuid: "b3e24b2b75be48a6b602b3085da5dba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3610a36c244fd2a23daf995f4ad3b0"
I20260929 16:46:32.660161 18662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4b22124b4f24384b3a15373b6ade7cd" candidate_uuid: "b3e24b2b75be48a6b602b3085da5dba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7f9427b9d964322bd42454d632b5394"
I20260929 16:46:32.660394 18662 raft_consensus.cc:3037] T d4b22124b4f24384b3a15373b6ade7cd P c7f9427b9d964322bd42454d632b5394 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:46:32.660382 18793 raft_consensus.cc:3037] T d4b22124b4f24384b3a15373b6ade7cd P da3610a36c244fd2a23daf995f4ad3b0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:46:32.662811 18793 raft_consensus.cc:2454] T d4b22124b4f24384b3a15373b6ade7cd P da3610a36c244fd2a23daf995f4ad3b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3e24b2b75be48a6b602b3085da5dba3 in term 1.
I20260929 16:46:32.662936 18662 raft_consensus.cc:2454] T d4b22124b4f24384b3a15373b6ade7cd P c7f9427b9d964322bd42454d632b5394 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3e24b2b75be48a6b602b3085da5dba3 in term 1.
I20260929 16:46:32.663451 18858 leader_election.cc:302] T d4b22124b4f24384b3a15373b6ade7cd P b3e24b2b75be48a6b602b3085da5dba3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3e24b2b75be48a6b602b3085da5dba3, da3610a36c244fd2a23daf995f4ad3b0; no voters:
I20260929 16:46:32.663825 18994 raft_consensus.cc:2781] T d4b22124b4f24384b3a15373b6ade7cd P b3e24b2b75be48a6b602b3085da5dba3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:46:32.664294 18994 raft_consensus.cc:686] T d4b22124b4f24384b3a15373b6ade7cd P b3e24b2b75be48a6b602b3085da5dba3 [term 1 LEADER]: Becoming Leader. State: Replica: b3e24b2b75be48a6b602b3085da5dba3, State: Running, Role: LEADER
I20260929 16:46:32.664834 18994 consensus_queue.cc:237] T d4b22124b4f24384b3a15373b6ade7cd P b3e24b2b75be48a6b602b3085da5dba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e24b2b75be48a6b602b3085da5dba3" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46627 } } peers { permanent_uuid: "c7f9427b9d964322bd42454d632b5394" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 45283 } } peers { permanent_uuid: "da3610a36c244fd2a23daf995f4ad3b0" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 45429 } }
I20260929 16:46:32.669891 18521 catalog_manager.cc:5495] T d4b22124b4f24384b3a15373b6ade7cd P b3e24b2b75be48a6b602b3085da5dba3 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3e24b2b75be48a6b602b3085da5dba3 (127.0.197.67). New cstate: current_term: 1 leader_uuid: "b3e24b2b75be48a6b602b3085da5dba3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e24b2b75be48a6b602b3085da5dba3" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 46627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c7f9427b9d964322bd42454d632b5394" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 45283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da3610a36c244fd2a23daf995f4ad3b0" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 45429 } health_report { overall_health: UNKNOWN } } }
I20260929 16:46:32.718215 19004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:32.711685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54896 (local address 127.0.197.67:46627)
0929 16:46:32.712444 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:32.712449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:32.712495 (+ 46us) server_negotiation.cc:407] Connection header received
0929 16:46:32.712730 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:32.712734 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:32.712824 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:32.713043 (+ 219us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:46:32.713480 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:32.714315 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:32.715870 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:32.716302 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:32.716403 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:32.716933 (+ 530us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:46:32.717484 (+ 551us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:46:32.717716 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:32.717886 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 16:46:32.717970 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":598,"threads_started":1}
I20260929 16:46:32.727172 18793 raft_consensus.cc:1261] T d4b22124b4f24384b3a15373b6ade7cd P da3610a36c244fd2a23daf995f4ad3b0 [term 1 FOLLOWER]: Refusing update from remote peer b3e24b2b75be48a6b602b3085da5dba3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:46:32.727172 18662 raft_consensus.cc:1261] T d4b22124b4f24384b3a15373b6ade7cd P c7f9427b9d964322bd42454d632b5394 [term 1 FOLLOWER]: Refusing update from remote peer b3e24b2b75be48a6b602b3085da5dba3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:46:32.728304 18994 consensus_queue.cc:1035] T d4b22124b4f24384b3a15373b6ade7cd P b3e24b2b75be48a6b602b3085da5dba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3610a36c244fd2a23daf995f4ad3b0" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 45429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:46:32.728679 18997 consensus_queue.cc:1035] T d4b22124b4f24384b3a15373b6ade7cd P b3e24b2b75be48a6b602b3085da5dba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7f9427b9d964322bd42454d632b5394" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 45283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:46:32.740438 19005 mvcc.cc:205] Tried to move back new op lower bound from 7334708808599019520 to 7334708808359079936. Current Snapshot: MvccSnapshot[applied={T|T < 7334708808599019520}]
I20260929 16:46:32.743423 19008 mvcc.cc:205] Tried to move back new op lower bound from 7334708808599019520 to 7334708808359079936. Current Snapshot: MvccSnapshot[applied={T|T < 7334708808599019520}]
I20260929 16:46:32.760317 18521 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:46:32.760635 18521 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:46:32.764403 18521 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 855457d956d64030ae26058c8ce2a8d8: Sending DeleteTablet for 3 replicas of tablet d4b22124b4f24384b3a15373b6ade7cd
I20260929 16:46:32.765717 18903 tablet_service.cc:1514] Processing DeleteTablet for tablet d4b22124b4f24384b3a15373b6ade7cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:46:32 UTC) from {username='slave'} at 127.0.0.1:54882
I20260929 16:46:32.765756 18773 tablet_service.cc:1514] Processing DeleteTablet for tablet d4b22124b4f24384b3a15373b6ade7cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:46:32 UTC) from {username='slave'} at 127.0.0.1:57610
I20260929 16:46:32.765789 18642 tablet_service.cc:1514] Processing DeleteTablet for tablet d4b22124b4f24384b3a15373b6ade7cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:46:32 UTC) from {username='slave'} at 127.0.0.1:56442
I20260929 16:46:32.766857 19018 tablet_replica.cc:330] T d4b22124b4f24384b3a15373b6ade7cd P c7f9427b9d964322bd42454d632b5394: stopping tablet replica
I20260929 16:46:32.767453 19018 raft_consensus.cc:2229] T d4b22124b4f24384b3a15373b6ade7cd P c7f9427b9d964322bd42454d632b5394 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:46:32.768102 19018 raft_consensus.cc:2258] T d4b22124b4f24384b3a15373b6ade7cd P c7f9427b9d964322bd42454d632b5394 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:46:32.770421 19018 ts_tablet_manager.cc:1905] T d4b22124b4f24384b3a15373b6ade7cd P c7f9427b9d964322bd42454d632b5394: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:46:32.773593 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 18580
I20260929 16:46:32.781315 19017 tablet_replica.cc:330] T d4b22124b4f24384b3a15373b6ade7cd P da3610a36c244fd2a23daf995f4ad3b0: stopping tablet replica
I20260929 16:46:32.782024 19017 raft_consensus.cc:2229] T d4b22124b4f24384b3a15373b6ade7cd P da3610a36c244fd2a23daf995f4ad3b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:46:32.782725 19017 raft_consensus.cc:2258] T d4b22124b4f24384b3a15373b6ade7cd P da3610a36c244fd2a23daf995f4ad3b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:46:32.783865 19019 tablet_replica.cc:330] T d4b22124b4f24384b3a15373b6ade7cd P b3e24b2b75be48a6b602b3085da5dba3: stopping tablet replica
I20260929 16:46:32.784456 19019 raft_consensus.cc:2229] T d4b22124b4f24384b3a15373b6ade7cd P b3e24b2b75be48a6b602b3085da5dba3 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:46:32.785334 19019 raft_consensus.cc:2258] T d4b22124b4f24384b3a15373b6ade7cd P b3e24b2b75be48a6b602b3085da5dba3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:46:32.788190 19019 ts_tablet_manager.cc:1905] T d4b22124b4f24384b3a15373b6ade7cd P b3e24b2b75be48a6b602b3085da5dba3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:46:32.789528 19018 ts_tablet_manager.cc:1918] T d4b22124b4f24384b3a15373b6ade7cd P c7f9427b9d964322bd42454d632b5394: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:46:32.789806 19018 log.cc:1196] T d4b22124b4f24384b3a15373b6ade7cd P c7f9427b9d964322bd42454d632b5394: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/ts-0/wal/wals/d4b22124b4f24384b3a15373b6ade7cd
I20260929 16:46:32.790501 19018 ts_tablet_manager.cc:1939] T d4b22124b4f24384b3a15373b6ade7cd P c7f9427b9d964322bd42454d632b5394: Deleting consensus metadata
I20260929 16:46:32.792776 18505 catalog_manager.cc:4841] TS c7f9427b9d964322bd42454d632b5394 (127.0.197.65:45283): tablet d4b22124b4f24384b3a15373b6ade7cd (table test-table [id=7a8164dd8aff4103a32f5353e767654d]) successfully deleted
I20260929 16:46:32.794149 19017 ts_tablet_manager.cc:1905] T d4b22124b4f24384b3a15373b6ade7cd P da3610a36c244fd2a23daf995f4ad3b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:46:32.794461 18985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:32.775705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56456 (local address 127.0.197.65:45283)
0929 16:46:32.775968 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:32.775978 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:32.786031 (+ 10053us) server_negotiation.cc:407] Connection header received
0929 16:46:32.786123 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:32.786128 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:32.786237 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:32.786363 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:32.786846 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:32.788086 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:32.790156 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:32.790881 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:32.790998 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:32.792646 (+ 1648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:32.792658 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:32.792676 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:46:32.793502 (+ 826us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:32.793571 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:46:32.793606 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:32.793727 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:32.793951 (+ 224us) server_negotiation.cc:299] Negotiation successful
0929 16:46:32.794079 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 16:46:32.795612 18702 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:46:32.798414 19019 ts_tablet_manager.cc:1918] T d4b22124b4f24384b3a15373b6ade7cd P b3e24b2b75be48a6b602b3085da5dba3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:46:32.798713 19019 log.cc:1196] T d4b22124b4f24384b3a15373b6ade7cd P b3e24b2b75be48a6b602b3085da5dba3: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/ts-2/wal/wals/d4b22124b4f24384b3a15373b6ade7cd
I20260929 16:46:32.799522 19019 ts_tablet_manager.cc:1939] T d4b22124b4f24384b3a15373b6ade7cd P b3e24b2b75be48a6b602b3085da5dba3: Deleting consensus metadata
I20260929 16:46:32.801667 18508 catalog_manager.cc:4841] TS b3e24b2b75be48a6b602b3085da5dba3 (127.0.197.67:46627): tablet d4b22124b4f24384b3a15373b6ade7cd (table test-table [id=7a8164dd8aff4103a32f5353e767654d]) successfully deleted
I20260929 16:46:32.806334 19017 ts_tablet_manager.cc:1918] T d4b22124b4f24384b3a15373b6ade7cd P da3610a36c244fd2a23daf995f4ad3b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:46:32.806577 19017 log.cc:1196] T d4b22124b4f24384b3a15373b6ade7cd P da3610a36c244fd2a23daf995f4ad3b0: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790700068255611-789-0/minicluster-data/ts-1/wal/wals/d4b22124b4f24384b3a15373b6ade7cd
I20260929 16:46:32.807104 19017 ts_tablet_manager.cc:1939] T d4b22124b4f24384b3a15373b6ade7cd P da3610a36c244fd2a23daf995f4ad3b0: Deleting consensus metadata
I20260929 16:46:32.808835 18507 catalog_manager.cc:4841] TS da3610a36c244fd2a23daf995f4ad3b0 (127.0.197.66:45429): tablet d4b22124b4f24384b3a15373b6ade7cd (table test-table [id=7a8164dd8aff4103a32f5353e767654d]) successfully deleted
I20260929 16:46:32.924418 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 18580
I20260929 16:46:32.943281 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 18711
I20260929 16:46:32.951524 18984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:32.944168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57614 (local address 127.0.197.66:45429)
0929 16:46:32.944330 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:32.944335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:32.944354 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:46:32.944558 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:32.944562 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:32.944666 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:32.944756 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:32.945220 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:32.945945 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:32.948062 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:32.948537 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:32.948651 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:32.949874 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:32.949884 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:32.949900 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:46:32.950583 (+ 683us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:32.950654 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:46:32.950676 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:32.950774 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:32.950940 (+ 166us) server_negotiation.cc:299] Negotiation successful
0929 16:46:32.951139 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 16:46:32.952351 18833 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:46:33.048195 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 18711
I20260929 16:46:33.066886 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 18841
I20260929 16:46:33.075902 19004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:33.067857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54904 (local address 127.0.197.67:46627)
0929 16:46:33.068079 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:33.068084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:33.068211 (+ 127us) server_negotiation.cc:407] Connection header received
0929 16:46:33.068407 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:33.068411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:33.068498 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:33.068594 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:33.069121 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:33.069861 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:33.071972 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:33.072463 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:33.072572 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:33.073979 (+ 1407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:33.073992 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:33.074007 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:46:33.074682 (+ 675us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:33.074743 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:46:33.074766 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:33.074868 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:33.075484 (+ 616us) server_negotiation.cc:299] Negotiation successful
0929 16:46:33.075606 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 16:46:33.076901 18963 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:46:33.201582 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 18841
I20260929 16:46:33.223237 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 18490
I20260929 16:46:33.224435 18551 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:46:33.337764 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 18490
2026-09-29T16:46:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3409 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:46:33 dist-test-slave-l31f krb5kdc[19027](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:46:33 dist-test-slave-l31f krb5kdc[19027](info): set up 2 sockets
Sep 29 16:46:33 dist-test-slave-l31f krb5kdc[19027](info): commencing operation
krb5kdc: starting...
W20260929 16:46:34.459348 789 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/krb5kdc/test-user.keytab.
Sep 29 16:46:34 dist-test-slave-l31f krb5kdc[19027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:46:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:46:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.197.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
I20260929 16:46:34.730121 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:39607
--webserver_interface=127.0.197.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:39773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.126:39607
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:46:35.088136 19040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:46:35.088464 19040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:46:35.088632 19040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:46:35.097029 19040 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:46:35.097189 19040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:46:35.097289 19040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:46:35.097404 19040 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:46:35.097487 19040 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:46:35.108191 19040 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:39773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.197.126:39607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:39607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.197.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:46:35.109655 19040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:46:35.111155 19040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:46:35.125021 19045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:35.125010 19046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:35.125049 19048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:46:35.126065 19040 server_base.cc:1034] running on GCE node
I20260929 16:46:35.127065 19040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:46:35.128773 19040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:46:35.130074 19040 hybrid_clock.cc:648] HybridClock initialized: now 1790700395130020 us; error 40 us; skew 500 ppm
I20260929 16:46:35.130564 19040 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:46:35 dist-test-slave-l31f krb5kdc[19027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:46:35.134318 19040 init.cc:377] Logged in from keytab as kudu/127.0.197.126@KRBTEST.COM (short username kudu)
I20260929 16:46:35.136307 19040 webserver.cc:460] Webserver started at http://127.0.197.126:35671/ using document root <none> and password file <none>
I20260929 16:46:35.137106 19040 fs_manager.cc:359] Metadata directory not provided
I20260929 16:46:35.137279 19040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:46:35.137650 19040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:46:35.141052 19040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/master-0/data/instance:
uuid: "7e63cd66df9f4909abd8f00da1419737"
format_stamp: "Formatted at 2026-09-29 16:46:35 on dist-test-slave-l31f"
server_key: "04050c02700ab6b9c62da0fafd1dd542"
server_key_iv: "7b65db06f0932162ded2fe430abd5468"
server_key_version: "encryptionkey@0"
I20260929 16:46:35.141969 19040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/master-0/wal/instance:
uuid: "7e63cd66df9f4909abd8f00da1419737"
format_stamp: "Formatted at 2026-09-29 16:46:35 on dist-test-slave-l31f"
server_key: "04050c02700ab6b9c62da0fafd1dd542"
server_key_iv: "7b65db06f0932162ded2fe430abd5468"
server_key_version: "encryptionkey@0"
I20260929 16:46:35.148684 19040 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260929 16:46:35.152782 19055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:35.154512 19040 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:46:35.155121 19040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "7e63cd66df9f4909abd8f00da1419737"
format_stamp: "Formatted at 2026-09-29 16:46:35 on dist-test-slave-l31f"
server_key: "04050c02700ab6b9c62da0fafd1dd542"
server_key_iv: "7b65db06f0932162ded2fe430abd5468"
server_key_version: "encryptionkey@0"
I20260929 16:46:35.155558 19040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:46:35.181137 19040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:46:35.186245 19040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:46:35.186765 19040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:46:35.208312 19040 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.126:39607
I20260929 16:46:35.208330 19106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.126:39607 every 8 connection(s)
I20260929 16:46:35.209916 19040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:46:35.214933 19107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:46:35.217476 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 19040
I20260929 16:46:35.217723 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/master-0/wal/instance
I20260929 16:46:35.218204 789 external_mini_cluster.cc:1428] Setting key 2e2f26285a209c93ec078ad0d737ff68
I20260929 16:46:35.231172 19107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e63cd66df9f4909abd8f00da1419737: Bootstrap starting.
Sep 29 16:46:35 dist-test-slave-l31f krb5kdc[19027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:46:35.235941 19107 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e63cd66df9f4909abd8f00da1419737: Neither blocks nor log segments found. Creating new log.
I20260929 16:46:35.237165 19107 log.cc:824] T 00000000000000000000000000000000 P 7e63cd66df9f4909abd8f00da1419737: Log is configured to *not* fsync() on all Append() calls
I20260929 16:46:35.240765 19107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e63cd66df9f4909abd8f00da1419737: No bootstrap required, opened a new log
I20260929 16:46:35.245857 19110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:35.220537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55972 (local address 127.0.197.126:39607)
0929 16:46:35.221494 (+ 957us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:35.221518 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:35.221675 (+ 157us) server_negotiation.cc:407] Connection header received
0929 16:46:35.224340 (+ 2665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:35.224377 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:35.224967 (+ 590us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:35.225477 (+ 510us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:35.227171 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:35.228798 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:35.230229 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:35.231109 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:35.231344 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:35.236294 (+ 4950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:35.236304 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:35.236322 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:46:35.236380 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:35.239924 (+ 3544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:35.241063 (+ 1139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:35.241075 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:35.241085 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:35.241341 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:35.241833 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:35.241838 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:35.241842 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:35.242603 (+ 761us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:35.243041 (+ 438us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:35.243670 (+ 629us) server_negotiation.cc:299] Negotiation successful
0929 16:46:35.244243 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":398,"threads_started":1}
I20260929 16:46:35.251721 19107 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e63cd66df9f4909abd8f00da1419737 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e63cd66df9f4909abd8f00da1419737" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 39607 } }
I20260929 16:46:35.252179 19107 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e63cd66df9f4909abd8f00da1419737 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:46:35.252310 19107 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e63cd66df9f4909abd8f00da1419737 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e63cd66df9f4909abd8f00da1419737, State: Initialized, Role: FOLLOWER
I20260929 16:46:35.252996 19107 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e63cd66df9f4909abd8f00da1419737 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e63cd66df9f4909abd8f00da1419737" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 39607 } }
I20260929 16:46:35.253243 19107 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e63cd66df9f4909abd8f00da1419737 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:46:35.253401 19107 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e63cd66df9f4909abd8f00da1419737 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:46:35.253589 19107 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e63cd66df9f4909abd8f00da1419737 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:46:35.256497 19107 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e63cd66df9f4909abd8f00da1419737 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e63cd66df9f4909abd8f00da1419737" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 39607 } }
I20260929 16:46:35.257113 19107 leader_election.cc:302] T 00000000000000000000000000000000 P 7e63cd66df9f4909abd8f00da1419737 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e63cd66df9f4909abd8f00da1419737; no voters:
I20260929 16:46:35.257957 19107 leader_election.cc:288] T 00000000000000000000000000000000 P 7e63cd66df9f4909abd8f00da1419737 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:46:35.258163 19112 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e63cd66df9f4909abd8f00da1419737 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:46:35.258903 19112 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e63cd66df9f4909abd8f00da1419737 [term 1 LEADER]: Becoming Leader. State: Replica: 7e63cd66df9f4909abd8f00da1419737, State: Running, Role: LEADER
I20260929 16:46:35.259605 19112 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e63cd66df9f4909abd8f00da1419737 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e63cd66df9f4909abd8f00da1419737" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 39607 } }
I20260929 16:46:35.260313 19107 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e63cd66df9f4909abd8f00da1419737 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:46:35.262863 19113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e63cd66df9f4909abd8f00da1419737 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e63cd66df9f4909abd8f00da1419737" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e63cd66df9f4909abd8f00da1419737" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 39607 } } }
I20260929 16:46:35.263419 19113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e63cd66df9f4909abd8f00da1419737 [sys.catalog]: This master's current role is: LEADER
I20260929 16:46:35.262863 19114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e63cd66df9f4909abd8f00da1419737 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e63cd66df9f4909abd8f00da1419737. Latest consensus state: current_term: 1 leader_uuid: "7e63cd66df9f4909abd8f00da1419737" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e63cd66df9f4909abd8f00da1419737" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 39607 } } }
I20260929 16:46:35.265369 19114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e63cd66df9f4909abd8f00da1419737 [sys.catalog]: This master's current role is: LEADER
I20260929 16:46:35.267689 19119 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:46:35.274050 19119 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:46:35.284822 19119 catalog_manager.cc:1347] Generated new cluster ID: 38fc28c3246e4efaab08f093ae89879a
I20260929 16:46:35.285000 19119 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:46:35.315721 19119 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:46:35.317868 19119 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:46:35.332435 19119 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e63cd66df9f4909abd8f00da1419737: Generated new TSK 0
I20260929 16:46:35.333403 19119 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.197.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab.
Entry for principal kudu/127.0.197.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab.
I20260929 16:46:35.423744 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.65:0
--local_ip_for_outbound_sockets=127.0.197.65
--webserver_interface=127.0.197.65
--webserver_port=0
--tserver_master_addrs=127.0.197.126:39607
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab
--principal=kudu/127.0.197.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:39773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:46:35.794116 19133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:46:35.794466 19133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:46:35.794655 19133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:46:35.802273 19133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:46:35.802420 19133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:46:35.802614 19133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.65
I20260929 16:46:35.813820 19133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:39773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab
--principal=kudu/127.0.197.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.197.65
--webserver_port=0
--tserver_master_addrs=127.0.197.126:39607
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.65
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:46:35.815393 19133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:46:35.816819 19133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:46:35.831873 19138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:35.831915 19141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:35.832666 19139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:46:35.833047 19133 server_base.cc:1034] running on GCE node
I20260929 16:46:35.833729 19133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:46:35.835134 19133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:46:35.836385 19133 hybrid_clock.cc:648] HybridClock initialized: now 1790700395836337 us; error 63 us; skew 500 ppm
I20260929 16:46:35.836745 19133 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:46:35 dist-test-slave-l31f krb5kdc[19027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:46:35.845489 19133 init.cc:377] Logged in from keytab as kudu/127.0.197.65@KRBTEST.COM (short username kudu)
I20260929 16:46:35.847895 19133 webserver.cc:460] Webserver started at http://127.0.197.65:33781/ using document root <none> and password file <none>
I20260929 16:46:35.848685 19133 fs_manager.cc:359] Metadata directory not provided
I20260929 16:46:35.848807 19133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:46:35.849113 19133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:46:35.852421 19133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/ts-0/data/instance:
uuid: "213bb71d8ab74a54ade6481abfe6ad00"
format_stamp: "Formatted at 2026-09-29 16:46:35 on dist-test-slave-l31f"
server_key: "f39403c34bbfab83f5211ac86b2592b6"
server_key_iv: "f1c5160dfb2b1cd6623ff72e64c7355f"
server_key_version: "encryptionkey@0"
I20260929 16:46:35.853314 19133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/ts-0/wal/instance:
uuid: "213bb71d8ab74a54ade6481abfe6ad00"
format_stamp: "Formatted at 2026-09-29 16:46:35 on dist-test-slave-l31f"
server_key: "f39403c34bbfab83f5211ac86b2592b6"
server_key_iv: "f1c5160dfb2b1cd6623ff72e64c7355f"
server_key_version: "encryptionkey@0"
I20260929 16:46:35.859189 19133 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260929 16:46:35.863502 19148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:35.865259 19133 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 16:46:35.865474 19133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/ts-0/wal
uuid: "213bb71d8ab74a54ade6481abfe6ad00"
format_stamp: "Formatted at 2026-09-29 16:46:35 on dist-test-slave-l31f"
server_key: "f39403c34bbfab83f5211ac86b2592b6"
server_key_iv: "f1c5160dfb2b1cd6623ff72e64c7355f"
server_key_version: "encryptionkey@0"
I20260929 16:46:35.865721 19133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:46:35.889605 19133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:46:35.894737 19133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:46:35.895109 19133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:46:35.896194 19133 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:46:35.897670 19133 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:46:35.897753 19133 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:35.897887 19133 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:46:35.897970 19133 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:35.933182 19133 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.65:37499
I20260929 16:46:35.933243 19260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.65:37499 every 8 connection(s)
I20260929 16:46:35.934579 19133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
I20260929 16:46:35.935928 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 19133
I20260929 16:46:35.936112 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/ts-0/wal/instance
I20260929 16:46:35.936669 789 external_mini_cluster.cc:1428] Setting key d9be29e9619581a9df0b30e2410fb89c
Sep 29 16:46:35 dist-test-slave-l31f krb5kdc[19027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.65@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:46:35.958016 19263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:35.937667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:35415 (local address 127.0.197.126:39607)
0929 16:46:35.938185 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:35.938189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:35.939512 (+ 1323us) server_negotiation.cc:407] Connection header received
0929 16:46:35.941610 (+ 2098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:35.941619 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:35.941730 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:35.941854 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:35.943578 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:35.944758 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:35.946469 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:35.946861 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:35.946975 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:35.951659 (+ 4684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:35.951665 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:35.951668 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:46:35.951707 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:35.954425 (+ 2718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:35.955401 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:35.955405 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:35.955408 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:35.955541 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:35.956035 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:35.956039 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:35.956041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:35.956334 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:35.956506 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:35.957381 (+ 875us) server_negotiation.cc:299] Negotiation successful
0929 16:46:35.957651 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":285,"threads_started":1}
I20260929 16:46:35.959183 19264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:35.938380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:39607 (local address 127.0.197.65:35415)
0929 16:46:35.939327 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:35.939375 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 16:46:35.941315 (+ 1940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:46:35.942099 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:46:35.942109 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:46:35.942654 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:46:35.943366 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:35.943387 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:35.944962 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:35.944967 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:35.946260 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:35.946274 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:35.947210 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:35.947221 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:35.947571 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:35.948360 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:46:35.948395 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:46:35.951354 (+ 2959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:46:35.954615 (+ 3261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:46:35.954626 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:46:35.954647 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:46:35.955244 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:46:35.955674 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:46:35.955679 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:46:35.955683 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:46:35.955896 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:46:35.956479 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:46:35.956488 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:46:35.956884 (+ 396us) client_negotiation.cc:769] Sending connection context
0929 16:46:35.957247 (+ 363us) client_negotiation.cc:240] Negotiation successful
0929 16:46:35.957840 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":380,"threads_started":1}
I20260929 16:46:35.961265 19261 heartbeater.cc:347] Connected to a master server at 127.0.197.126:39607
I20260929 16:46:35.961683 19261 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.197.66@KRBTEST.COM; defaulting to no policy
I20260929 16:46:35.962419 19261 heartbeater.cc:511] Master 127.0.197.126:39607 requested a full tablet report, sending...
I20260929 16:46:35.965206 19072 ts_manager.cc:194] Registered new tserver with Master: 213bb71d8ab74a54ade6481abfe6ad00 (127.0.197.65:37499)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.66@KRBTEST.COM" created.
I20260929 16:46:35.966854 19072 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.65@KRBTEST.COM'} at 127.0.197.65:35415
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab.
Entry for principal kudu/127.0.197.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab.
I20260929 16:46:36.000492 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.66:0
--local_ip_for_outbound_sockets=127.0.197.66
--webserver_interface=127.0.197.66
--webserver_port=0
--tserver_master_addrs=127.0.197.126:39607
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab
--principal=kudu/127.0.197.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:39773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:46:36.356472 19267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:46:36.356787 19267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:46:36.356887 19267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:46:36.365303 19267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:46:36.365443 19267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:46:36.365563 19267 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.66
I20260929 16:46:36.376524 19267 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:39773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab
--principal=kudu/127.0.197.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.197.66
--webserver_port=0
--tserver_master_addrs=127.0.197.126:39607
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.66
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:46:36.377880 19267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:46:36.379328 19267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:46:36.394105 19272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:36.394038 19275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:36.394119 19273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:46:36.394845 19267 server_base.cc:1034] running on GCE node
I20260929 16:46:36.395579 19267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:46:36.396665 19267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:46:36.397928 19267 hybrid_clock.cc:648] HybridClock initialized: now 1790700396397870 us; error 42 us; skew 500 ppm
I20260929 16:46:36.398330 19267 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:46:36 dist-test-slave-l31f krb5kdc[19027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700396, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:46:36.401849 19267 init.cc:377] Logged in from keytab as kudu/127.0.197.66@KRBTEST.COM (short username kudu)
I20260929 16:46:36.403714 19267 webserver.cc:460] Webserver started at http://127.0.197.66:44839/ using document root <none> and password file <none>
I20260929 16:46:36.404489 19267 fs_manager.cc:359] Metadata directory not provided
I20260929 16:46:36.404666 19267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:46:36.405004 19267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:46:36.408310 19267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/ts-1/data/instance:
uuid: "057527a7327b4978babf4adcb767babb"
format_stamp: "Formatted at 2026-09-29 16:46:36 on dist-test-slave-l31f"
server_key: "74461830453052a7658b046933a947fe"
server_key_iv: "55b70fedaf85a6a413a89f41b12f47a0"
server_key_version: "encryptionkey@0"
I20260929 16:46:36.409229 19267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/ts-1/wal/instance:
uuid: "057527a7327b4978babf4adcb767babb"
format_stamp: "Formatted at 2026-09-29 16:46:36 on dist-test-slave-l31f"
server_key: "74461830453052a7658b046933a947fe"
server_key_iv: "55b70fedaf85a6a413a89f41b12f47a0"
server_key_version: "encryptionkey@0"
I20260929 16:46:36.415244 19267 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 16:46:36.419744 19282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:36.421341 19267 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 16:46:36.421573 19267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/ts-1/wal
uuid: "057527a7327b4978babf4adcb767babb"
format_stamp: "Formatted at 2026-09-29 16:46:36 on dist-test-slave-l31f"
server_key: "74461830453052a7658b046933a947fe"
server_key_iv: "55b70fedaf85a6a413a89f41b12f47a0"
server_key_version: "encryptionkey@0"
I20260929 16:46:36.421820 19267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:46:36.438177 19267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:46:36.442788 19267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:46:36.443220 19267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:46:36.444360 19267 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:46:36.445860 19267 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:46:36.445972 19267 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:36.446130 19267 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:46:36.446208 19267 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:36.483539 19267 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.66:38157
I20260929 16:46:36.483582 19394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.66:38157 every 8 connection(s)
I20260929 16:46:36.485036 19267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
I20260929 16:46:36.487725 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 19267
I20260929 16:46:36.487939 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/ts-1/wal/instance
I20260929 16:46:36.488539 789 external_mini_cluster.cc:1428] Setting key 5e6c321a6f1a788d4fa12e4319836dd4
Sep 29 16:46:36 dist-test-slave-l31f krb5kdc[19027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700396, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.66@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:46:36.508256 19397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:36.487948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:47031 (local address 127.0.197.126:39607)
0929 16:46:36.488616 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:36.488623 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:36.489973 (+ 1350us) server_negotiation.cc:407] Connection header received
0929 16:46:36.492180 (+ 2207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:36.492186 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:36.492287 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:36.492429 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:36.494171 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:36.495075 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:36.496891 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:36.497382 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:36.497536 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:36.501818 (+ 4282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:36.501824 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:36.501828 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:46:36.501865 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:36.504781 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:36.505679 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:36.505685 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:36.505689 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:36.505833 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:36.506312 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:36.506317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:36.506321 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:36.506633 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:36.506797 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:36.507622 (+ 825us) server_negotiation.cc:299] Negotiation successful
0929 16:46:36.507874 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":405,"threads_started":1}
I20260929 16:46:36.509356 19398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:36.488822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:39607 (local address 127.0.197.66:47031)
0929 16:46:36.489774 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:36.489840 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 16:46:36.491868 (+ 2028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:46:36.492706 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:46:36.492717 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:46:36.493306 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:46:36.493968 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:36.493987 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:36.495271 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:36.495275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:36.496710 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:36.496724 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:36.497656 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:36.497664 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:36.497937 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:36.498790 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:46:36.498817 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:46:36.501504 (+ 2687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:46:36.504996 (+ 3492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:46:36.505007 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:46:36.505034 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:46:36.505529 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:46:36.505965 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:46:36.505971 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:46:36.505975 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:46:36.506187 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:46:36.506771 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:46:36.506780 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:46:36.507244 (+ 464us) client_negotiation.cc:769] Sending connection context
0929 16:46:36.507529 (+ 285us) client_negotiation.cc:240] Negotiation successful
0929 16:46:36.508123 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":382,"threads_started":1}
I20260929 16:46:36.511042 19395 heartbeater.cc:347] Connected to a master server at 127.0.197.126:39607
I20260929 16:46:36.511370 19395 heartbeater.cc:464] Registering TS with master...
I20260929 16:46:36.512130 19395 heartbeater.cc:511] Master 127.0.197.126:39607 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.197.67@KRBTEST.COM; defaulting to no policy
I20260929 16:46:36.513782 19072 ts_manager.cc:194] Registered new tserver with Master: 057527a7327b4978babf4adcb767babb (127.0.197.66:38157)
I20260929 16:46:36.514762 19072 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.66@KRBTEST.COM'} at 127.0.197.66:47031
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab.
Entry for principal kudu/127.0.197.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab.
I20260929 16:46:36.546921 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.67:0
--local_ip_for_outbound_sockets=127.0.197.67
--webserver_interface=127.0.197.67
--webserver_port=0
--tserver_master_addrs=127.0.197.126:39607
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab
--principal=kudu/127.0.197.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:39773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:46:36.904969 19401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:46:36.905292 19401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:46:36.905390 19401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:46:36.912782 19401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:46:36.912906 19401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:46:36.913021 19401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.67
I20260929 16:46:36.923252 19401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:39773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab
--principal=kudu/127.0.197.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.197.67
--webserver_port=0
--tserver_master_addrs=127.0.197.126:39607
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.67
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:46:36.924638 19401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:46:36.926011 19401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:46:36.941897 19406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:36.941946 19407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:36.941926 19409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:46:36.943127 19401 server_base.cc:1034] running on GCE node
I20260929 16:46:36.943820 19401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:46:36.944900 19401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:46:36.946174 19401 hybrid_clock.cc:648] HybridClock initialized: now 1790700396946129 us; error 35 us; skew 500 ppm
I20260929 16:46:36.946556 19401 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:46:36 dist-test-slave-l31f krb5kdc[19027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700396, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:46:36.949893 19401 init.cc:377] Logged in from keytab as kudu/127.0.197.67@KRBTEST.COM (short username kudu)
I20260929 16:46:36.951558 19401 webserver.cc:460] Webserver started at http://127.0.197.67:41787/ using document root <none> and password file <none>
I20260929 16:46:36.952317 19401 fs_manager.cc:359] Metadata directory not provided
I20260929 16:46:36.952458 19401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:46:36.952812 19401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:46:36.956032 19401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/ts-2/data/instance:
uuid: "08ef2d53b3fe485586d1e76003807f15"
format_stamp: "Formatted at 2026-09-29 16:46:36 on dist-test-slave-l31f"
server_key: "97c92643dd68f09c6431cb8767b12c5f"
server_key_iv: "209e9aadd13925b70bc349eb2824b78a"
server_key_version: "encryptionkey@0"
I20260929 16:46:36.957221 19401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/ts-2/wal/instance:
uuid: "08ef2d53b3fe485586d1e76003807f15"
format_stamp: "Formatted at 2026-09-29 16:46:36 on dist-test-slave-l31f"
server_key: "97c92643dd68f09c6431cb8767b12c5f"
server_key_iv: "209e9aadd13925b70bc349eb2824b78a"
server_key_version: "encryptionkey@0"
I20260929 16:46:36.965353 19401 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.003s
I20260929 16:46:36.969707 19416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:36.970655 19261 heartbeater.cc:503] Master 127.0.197.126:39607 was elected leader, sending a full tablet report...
I20260929 16:46:36.971626 19401 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:46:36.971832 19401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/ts-2/wal
uuid: "08ef2d53b3fe485586d1e76003807f15"
format_stamp: "Formatted at 2026-09-29 16:46:36 on dist-test-slave-l31f"
server_key: "97c92643dd68f09c6431cb8767b12c5f"
server_key_iv: "209e9aadd13925b70bc349eb2824b78a"
server_key_version: "encryptionkey@0"
I20260929 16:46:36.972088 19401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:46:37.006345 19401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:46:37.011710 19401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:46:37.012071 19401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:46:37.013273 19401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:46:37.014849 19401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:46:37.014968 19401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:37.015172 19401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:46:37.015256 19401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:37.053210 19401 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.67:38473
I20260929 16:46:37.053251 19528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.67:38473 every 8 connection(s)
I20260929 16:46:37.054570 19401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
I20260929 16:46:37.054921 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 19401
I20260929 16:46:37.055135 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/ts-2/wal/instance
I20260929 16:46:37.055666 789 external_mini_cluster.cc:1428] Setting key bde30c69f742dab64e1be1ad4d9b0675
Sep 29 16:46:37 dist-test-slave-l31f krb5kdc[19027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700396, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.67@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:46:37.076781 19531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:37.057274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:51611 (local address 127.0.197.126:39607)
0929 16:46:37.057827 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:37.057833 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:37.058976 (+ 1143us) server_negotiation.cc:407] Connection header received
0929 16:46:37.061207 (+ 2231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:37.061213 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:37.061308 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:37.061441 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:37.063105 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:37.063907 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:37.065513 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:37.065934 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:37.066081 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:37.070310 (+ 4229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:37.070318 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:37.070323 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:46:37.070369 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:37.073474 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:37.074303 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:37.074307 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:37.074310 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:37.074438 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:37.074870 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:37.074874 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:37.074876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:37.075296 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:37.075427 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:37.076262 (+ 835us) server_negotiation.cc:299] Negotiation successful
0929 16:46:37.076492 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":273,"threads_started":1}
I20260929 16:46:37.077900 19532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:37.057964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:39607 (local address 127.0.197.67:51611)
0929 16:46:37.058807 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:37.058856 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 16:46:37.060874 (+ 2018us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:46:37.061704 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:46:37.061712 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:46:37.062269 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:46:37.062862 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:37.062900 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:37.064085 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:37.064091 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:37.065346 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:37.065359 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:37.066181 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:37.066191 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:37.066445 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:37.067237 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:46:37.067261 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:46:37.069910 (+ 2649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:46:37.073631 (+ 3721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:46:37.073639 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:46:37.073655 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:46:37.074173 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:46:37.074536 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:46:37.074539 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:46:37.074542 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:46:37.074732 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:46:37.075445 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:46:37.075452 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:46:37.075850 (+ 398us) client_negotiation.cc:769] Sending connection context
0929 16:46:37.076135 (+ 285us) client_negotiation.cc:240] Negotiation successful
0929 16:46:37.076671 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":316,"threads_started":1}
I20260929 16:46:37.079471 19529 heartbeater.cc:347] Connected to a master server at 127.0.197.126:39607
I20260929 16:46:37.079833 19529 heartbeater.cc:464] Registering TS with master...
I20260929 16:46:37.080629 19529 heartbeater.cc:511] Master 127.0.197.126:39607 requested a full tablet report, sending...
I20260929 16:46:37.082468 19072 ts_manager.cc:194] Registered new tserver with Master: 08ef2d53b3fe485586d1e76003807f15 (127.0.197.67:38473)
I20260929 16:46:37.083644 19072 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.67@KRBTEST.COM'} at 127.0.197.67:51611
I20260929 16:46:37.085299 789 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:46:37.104401 19531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:37.090183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55988 (local address 127.0.197.126:39607)
0929 16:46:37.090448 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:37.090455 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:37.090819 (+ 364us) server_negotiation.cc:407] Connection header received
0929 16:46:37.090956 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:37.090960 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:37.091118 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:37.091214 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:37.092792 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:37.093553 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:37.094980 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:37.095459 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:37.095573 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:37.098552 (+ 2979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:37.098558 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:37.098561 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:46:37.098601 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:37.101235 (+ 2634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:37.102142 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:37.102146 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:37.102149 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:37.102303 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:37.102750 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:37.102753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:37.102755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:37.103235 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:37.103452 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:37.103862 (+ 410us) server_negotiation.cc:299] Negotiation successful
0929 16:46:37.104025 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 16:46:37.108918 19072 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55988:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:46:37.115993 19072 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:46:37.142340 19542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:37.129497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:37499 (local address 127.0.0.1:44262)
0929 16:46:37.130441 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:37.130485 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 16:46:37.130678 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:46:37.132219 (+ 1541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:46:37.132231 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:46:37.132287 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:46:37.132850 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:37.132867 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:37.135156 (+ 2289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:37.135165 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:37.138177 (+ 3012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:37.138194 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:37.140929 (+ 2735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:37.140940 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:37.141320 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:37.141416 (+ 96us) client_negotiation.cc:769] Sending connection context
0929 16:46:37.141588 (+ 172us) client_negotiation.cc:240] Negotiation successful
0929 16:46:37.141890 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":1084,"threads_started":1}
I20260929 16:46:37.145809 19543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:37.129855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44262 (local address 127.0.197.65:37499)
0929 16:46:37.131407 (+ 1552us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:37.131417 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:37.131448 (+ 31us) server_negotiation.cc:407] Connection header received
0929 16:46:37.131564 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:37.131575 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:37.131872 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:37.133132 (+ 1260us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:46:37.133387 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:37.134903 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:37.138436 (+ 3533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:37.140563 (+ 2127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:37.144769 (+ 4206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:37.145082 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:37.145185 (+ 103us) server_negotiation.cc:299] Negotiation successful
0929 16:46:37.145409 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1359,"thread_start_us":1200,"threads_started":1}
I20260929 16:46:37.151978 19542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:37.130055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.67:38473 (local address 127.0.0.1:40072)
0929 16:46:37.142654 (+ 12599us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:37.142676 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:46:37.142804 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:46:37.144020 (+ 1216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:46:37.144028 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:46:37.144074 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:46:37.144403 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:37.144419 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:37.147176 (+ 2757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:37.147187 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:37.148837 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:37.148855 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:37.150923 (+ 2068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:37.150930 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:37.151192 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:37.151243 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 16:46:37.151327 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 16:46:37.151493 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12477,"mutex_wait_us":3,"spinlock_wait_cycles":38528,"thread_start_us":14279,"threads_started":1}
I20260929 16:46:37.152076 19196 tablet_service.cc:1467] Processing CreateTablet for tablet 5c20e4cdc2f84921a6e988e70b3a17ff (DEFAULT_TABLE table=test-table [id=09d17e6d06de4c2a9becc00b217c7cb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:46:37.153378 19545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:37.130748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40072 (local address 127.0.197.67:38473)
0929 16:46:37.137938 (+ 7190us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:37.137952 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:37.143117 (+ 5165us) server_negotiation.cc:407] Connection header received
0929 16:46:37.143351 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:37.143367 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:37.143716 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:37.144651 (+ 935us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:46:37.145045 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:37.146966 (+ 1921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:37.149054 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:37.150717 (+ 1663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:37.152471 (+ 1754us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:37.152739 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:37.152819 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 16:46:37.153013 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":877,"thread_start_us":709,"threads_started":1}
I20260929 16:46:37.155557 19196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c20e4cdc2f84921a6e988e70b3a17ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:46:37.158138 19544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:37.131452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.66:38157 (local address 127.0.0.1:49568)
0929 16:46:37.143994 (+ 12542us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:37.144026 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 16:46:37.144224 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:46:37.145770 (+ 1546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:46:37.145778 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:46:37.145814 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:46:37.146694 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:37.146737 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:37.148911 (+ 2174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:37.148919 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:37.153389 (+ 4470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:37.153448 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:37.157115 (+ 3667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:37.157130 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:37.157322 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:37.157368 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 16:46:37.157478 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 16:46:37.157651 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12151,"spinlock_wait_cycles":1792,"thread_start_us":14913,"threads_started":1}
I20260929 16:46:37.159164 19547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:37.131986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49568 (local address 127.0.197.66:38157)
0929 16:46:37.144663 (+ 12677us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:37.144674 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:37.144708 (+ 34us) server_negotiation.cc:407] Connection header received
0929 16:46:37.144810 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:37.144818 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:37.145110 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:37.145331 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:46:37.146971 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:37.148697 (+ 1726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:37.153747 (+ 5050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:37.156884 (+ 3137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:37.157053 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:37.157254 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:37.158552 (+ 1298us) server_negotiation.cc:299] Negotiation successful
0929 16:46:37.158755 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12473,"thread_start_us":13505,"threads_started":1}
I20260929 16:46:37.159090 19464 tablet_service.cc:1467] Processing CreateTablet for tablet 5c20e4cdc2f84921a6e988e70b3a17ff (DEFAULT_TABLE table=test-table [id=09d17e6d06de4c2a9becc00b217c7cb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:46:37.161125 19464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c20e4cdc2f84921a6e988e70b3a17ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:46:37.164279 19330 tablet_service.cc:1467] Processing CreateTablet for tablet 5c20e4cdc2f84921a6e988e70b3a17ff (DEFAULT_TABLE table=test-table [id=09d17e6d06de4c2a9becc00b217c7cb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:46:37.166003 19330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c20e4cdc2f84921a6e988e70b3a17ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:46:37.179859 19548 tablet_bootstrap.cc:492] T 5c20e4cdc2f84921a6e988e70b3a17ff P 213bb71d8ab74a54ade6481abfe6ad00: Bootstrap starting.
I20260929 16:46:37.184376 19548 tablet_bootstrap.cc:654] T 5c20e4cdc2f84921a6e988e70b3a17ff P 213bb71d8ab74a54ade6481abfe6ad00: Neither blocks nor log segments found. Creating new log.
I20260929 16:46:37.186762 19548 log.cc:824] T 5c20e4cdc2f84921a6e988e70b3a17ff P 213bb71d8ab74a54ade6481abfe6ad00: Log is configured to *not* fsync() on all Append() calls
I20260929 16:46:37.191809 19549 tablet_bootstrap.cc:492] T 5c20e4cdc2f84921a6e988e70b3a17ff P 08ef2d53b3fe485586d1e76003807f15: Bootstrap starting.
I20260929 16:46:37.194640 19551 tablet_bootstrap.cc:492] T 5c20e4cdc2f84921a6e988e70b3a17ff P 057527a7327b4978babf4adcb767babb: Bootstrap starting.
I20260929 16:46:37.196427 19549 tablet_bootstrap.cc:654] T 5c20e4cdc2f84921a6e988e70b3a17ff P 08ef2d53b3fe485586d1e76003807f15: Neither blocks nor log segments found. Creating new log.
I20260929 16:46:37.197897 19549 log.cc:824] T 5c20e4cdc2f84921a6e988e70b3a17ff P 08ef2d53b3fe485586d1e76003807f15: Log is configured to *not* fsync() on all Append() calls
I20260929 16:46:37.198210 19551 tablet_bootstrap.cc:654] T 5c20e4cdc2f84921a6e988e70b3a17ff P 057527a7327b4978babf4adcb767babb: Neither blocks nor log segments found. Creating new log.
I20260929 16:46:37.198854 19548 tablet_bootstrap.cc:492] T 5c20e4cdc2f84921a6e988e70b3a17ff P 213bb71d8ab74a54ade6481abfe6ad00: No bootstrap required, opened a new log
I20260929 16:46:37.199282 19548 ts_tablet_manager.cc:1397] T 5c20e4cdc2f84921a6e988e70b3a17ff P 213bb71d8ab74a54ade6481abfe6ad00: Time spent bootstrapping tablet: real 0.020s user 0.012s sys 0.005s
I20260929 16:46:37.199623 19551 log.cc:824] T 5c20e4cdc2f84921a6e988e70b3a17ff P 057527a7327b4978babf4adcb767babb: Log is configured to *not* fsync() on all Append() calls
I20260929 16:46:37.201427 19549 tablet_bootstrap.cc:492] T 5c20e4cdc2f84921a6e988e70b3a17ff P 08ef2d53b3fe485586d1e76003807f15: No bootstrap required, opened a new log
I20260929 16:46:37.201745 19549 ts_tablet_manager.cc:1397] T 5c20e4cdc2f84921a6e988e70b3a17ff P 08ef2d53b3fe485586d1e76003807f15: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260929 16:46:37.211174 19548 raft_consensus.cc:348] T 5c20e4cdc2f84921a6e988e70b3a17ff P 213bb71d8ab74a54ade6481abfe6ad00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "213bb71d8ab74a54ade6481abfe6ad00" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 37499 } } peers { permanent_uuid: "057527a7327b4978babf4adcb767babb" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 38157 } } peers { permanent_uuid: "08ef2d53b3fe485586d1e76003807f15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 38473 } }
I20260929 16:46:37.211680 19548 raft_consensus.cc:374] T 5c20e4cdc2f84921a6e988e70b3a17ff P 213bb71d8ab74a54ade6481abfe6ad00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:46:37.211789 19548 raft_consensus.cc:729] T 5c20e4cdc2f84921a6e988e70b3a17ff P 213bb71d8ab74a54ade6481abfe6ad00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 213bb71d8ab74a54ade6481abfe6ad00, State: Initialized, Role: FOLLOWER
I20260929 16:46:37.212510 19548 consensus_queue.cc:260] T 5c20e4cdc2f84921a6e988e70b3a17ff P 213bb71d8ab74a54ade6481abfe6ad00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "213bb71d8ab74a54ade6481abfe6ad00" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 37499 } } peers { permanent_uuid: "057527a7327b4978babf4adcb767babb" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 38157 } } peers { permanent_uuid: "08ef2d53b3fe485586d1e76003807f15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 38473 } }
I20260929 16:46:37.213996 19549 raft_consensus.cc:348] T 5c20e4cdc2f84921a6e988e70b3a17ff P 08ef2d53b3fe485586d1e76003807f15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "213bb71d8ab74a54ade6481abfe6ad00" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 37499 } } peers { permanent_uuid: "057527a7327b4978babf4adcb767babb" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 38157 } } peers { permanent_uuid: "08ef2d53b3fe485586d1e76003807f15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 38473 } }
I20260929 16:46:37.214356 19548 ts_tablet_manager.cc:1428] T 5c20e4cdc2f84921a6e988e70b3a17ff P 213bb71d8ab74a54ade6481abfe6ad00: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260929 16:46:37.214494 19549 raft_consensus.cc:374] T 5c20e4cdc2f84921a6e988e70b3a17ff P 08ef2d53b3fe485586d1e76003807f15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:46:37.214603 19549 raft_consensus.cc:729] T 5c20e4cdc2f84921a6e988e70b3a17ff P 08ef2d53b3fe485586d1e76003807f15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08ef2d53b3fe485586d1e76003807f15, State: Initialized, Role: FOLLOWER
I20260929 16:46:37.215420 19549 consensus_queue.cc:260] T 5c20e4cdc2f84921a6e988e70b3a17ff P 08ef2d53b3fe485586d1e76003807f15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "213bb71d8ab74a54ade6481abfe6ad00" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 37499 } } peers { permanent_uuid: "057527a7327b4978babf4adcb767babb" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 38157 } } peers { permanent_uuid: "08ef2d53b3fe485586d1e76003807f15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 38473 } }
I20260929 16:46:37.215968 19551 tablet_bootstrap.cc:492] T 5c20e4cdc2f84921a6e988e70b3a17ff P 057527a7327b4978babf4adcb767babb: No bootstrap required, opened a new log
I20260929 16:46:37.216293 19551 ts_tablet_manager.cc:1397] T 5c20e4cdc2f84921a6e988e70b3a17ff P 057527a7327b4978babf4adcb767babb: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.003s
I20260929 16:46:37.227417 19551 raft_consensus.cc:348] T 5c20e4cdc2f84921a6e988e70b3a17ff P 057527a7327b4978babf4adcb767babb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "213bb71d8ab74a54ade6481abfe6ad00" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 37499 } } peers { permanent_uuid: "057527a7327b4978babf4adcb767babb" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 38157 } } peers { permanent_uuid: "08ef2d53b3fe485586d1e76003807f15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 38473 } }
I20260929 16:46:37.227963 19551 raft_consensus.cc:374] T 5c20e4cdc2f84921a6e988e70b3a17ff P 057527a7327b4978babf4adcb767babb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:46:37.228492 19551 raft_consensus.cc:729] T 5c20e4cdc2f84921a6e988e70b3a17ff P 057527a7327b4978babf4adcb767babb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 057527a7327b4978babf4adcb767babb, State: Initialized, Role: FOLLOWER
I20260929 16:46:37.229257 19551 consensus_queue.cc:260] T 5c20e4cdc2f84921a6e988e70b3a17ff P 057527a7327b4978babf4adcb767babb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "213bb71d8ab74a54ade6481abfe6ad00" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 37499 } } peers { permanent_uuid: "057527a7327b4978babf4adcb767babb" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 38157 } } peers { permanent_uuid: "08ef2d53b3fe485586d1e76003807f15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 38473 } }
I20260929 16:46:37.230530 19395 heartbeater.cc:503] Master 127.0.197.126:39607 was elected leader, sending a full tablet report...
I20260929 16:46:37.231264 19551 ts_tablet_manager.cc:1428] T 5c20e4cdc2f84921a6e988e70b3a17ff P 057527a7327b4978babf4adcb767babb: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
I20260929 16:46:37.231287 19529 heartbeater.cc:503] Master 127.0.197.126:39607 was elected leader, sending a full tablet report...
I20260929 16:46:37.232371 19549 ts_tablet_manager.cc:1428] T 5c20e4cdc2f84921a6e988e70b3a17ff P 08ef2d53b3fe485586d1e76003807f15: Time spent starting tablet: real 0.030s user 0.011s sys 0.019s
W20260929 16:46:37.237313 19396 tablet.cc:2369] T 5c20e4cdc2f84921a6e988e70b3a17ff P 057527a7327b4978babf4adcb767babb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:46:37.306505 19530 tablet.cc:2369] T 5c20e4cdc2f84921a6e988e70b3a17ff P 08ef2d53b3fe485586d1e76003807f15: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:46:37.437553 19262 tablet.cc:2369] T 5c20e4cdc2f84921a6e988e70b3a17ff P 213bb71d8ab74a54ade6481abfe6ad00: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:46:37.449180 19555 raft_consensus.cc:482] T 5c20e4cdc2f84921a6e988e70b3a17ff P 08ef2d53b3fe485586d1e76003807f15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:46:37.449517 19555 raft_consensus.cc:504] T 5c20e4cdc2f84921a6e988e70b3a17ff P 08ef2d53b3fe485586d1e76003807f15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "213bb71d8ab74a54ade6481abfe6ad00" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 37499 } } peers { permanent_uuid: "057527a7327b4978babf4adcb767babb" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 38157 } } peers { permanent_uuid: "08ef2d53b3fe485586d1e76003807f15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 38473 } }
I20260929 16:46:37.451380 19555 leader_election.cc:288] T 5c20e4cdc2f84921a6e988e70b3a17ff P 08ef2d53b3fe485586d1e76003807f15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 213bb71d8ab74a54ade6481abfe6ad00 (127.0.197.65:37499), 057527a7327b4978babf4adcb767babb (127.0.197.66:38157)
I20260929 16:46:37.458825 19543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:37.451952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:51795 (local address 127.0.197.65:37499)
0929 16:46:37.452192 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:37.452199 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:37.452223 (+ 24us) server_negotiation.cc:407] Connection header received
0929 16:46:37.452444 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:37.452448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:37.452511 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:37.452620 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:46:37.453307 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:37.454298 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:37.456764 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:37.457911 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:37.458109 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:37.458201 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:37.458423 (+ 222us) server_negotiation.cc:299] Negotiation successful
0929 16:46:37.458565 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 16:46:37.458809 19532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:37.451811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:37499 (local address 127.0.197.67:51795)
0929 16:46:37.452055 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:37.452082 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 16:46:37.452248 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:46:37.452831 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:46:37.452838 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:46:37.452881 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:46:37.453089 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:37.453101 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:37.454478 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:37.454483 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:37.456539 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:37.456557 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:37.458087 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:37.458091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:37.458242 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:37.458280 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 16:46:37.458359 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 16:46:37.458467 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260929 16:46:37.459511 19547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:37.452548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:57981 (local address 127.0.197.66:38157)
0929 16:46:37.452792 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:37.452797 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:37.453236 (+ 439us) server_negotiation.cc:407] Connection header received
0929 16:46:37.453329 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:37.453334 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:37.453417 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:37.453531 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:46:37.454081 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:37.454924 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:37.457228 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:37.458494 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:37.458666 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:37.458737 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:37.459186 (+ 449us) server_negotiation.cc:299] Negotiation successful
0929 16:46:37.459326 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260929 16:46:37.460155 19557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:37.452246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.66:38157 (local address 127.0.197.67:57981)
0929 16:46:37.453072 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:37.453091 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:46:37.453228 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:46:37.453545 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:46:37.453548 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:46:37.453566 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:46:37.453909 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:37.453917 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:37.455232 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:37.455238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:37.457057 (+ 1819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:37.457068 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:37.458727 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:37.458732 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:37.458921 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:37.458961 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 16:46:37.459670 (+ 709us) client_negotiation.cc:240] Negotiation successful
0929 16:46:37.459834 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"thread_start_us":423,"threads_started":1}
I20260929 16:46:37.460155 19216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c20e4cdc2f84921a6e988e70b3a17ff" candidate_uuid: "08ef2d53b3fe485586d1e76003807f15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "213bb71d8ab74a54ade6481abfe6ad00" is_pre_election: true
I20260929 16:46:37.460832 19216 raft_consensus.cc:2454] T 5c20e4cdc2f84921a6e988e70b3a17ff P 213bb71d8ab74a54ade6481abfe6ad00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08ef2d53b3fe485586d1e76003807f15 in term 0.
I20260929 16:46:37.461627 19420 leader_election.cc:302] T 5c20e4cdc2f84921a6e988e70b3a17ff P 08ef2d53b3fe485586d1e76003807f15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08ef2d53b3fe485586d1e76003807f15, 213bb71d8ab74a54ade6481abfe6ad00; no voters:
I20260929 16:46:37.461627 19350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c20e4cdc2f84921a6e988e70b3a17ff" candidate_uuid: "08ef2d53b3fe485586d1e76003807f15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "057527a7327b4978babf4adcb767babb" is_pre_election: true
I20260929 16:46:37.462164 19555 raft_consensus.cc:2781] T 5c20e4cdc2f84921a6e988e70b3a17ff P 08ef2d53b3fe485586d1e76003807f15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:46:37.462369 19555 raft_consensus.cc:482] T 5c20e4cdc2f84921a6e988e70b3a17ff P 08ef2d53b3fe485586d1e76003807f15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:46:37.462296 19350 raft_consensus.cc:2454] T 5c20e4cdc2f84921a6e988e70b3a17ff P 057527a7327b4978babf4adcb767babb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08ef2d53b3fe485586d1e76003807f15 in term 0.
I20260929 16:46:37.462487 19555 raft_consensus.cc:3037] T 5c20e4cdc2f84921a6e988e70b3a17ff P 08ef2d53b3fe485586d1e76003807f15 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:46:37.464996 19555 raft_consensus.cc:504] T 5c20e4cdc2f84921a6e988e70b3a17ff P 08ef2d53b3fe485586d1e76003807f15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "213bb71d8ab74a54ade6481abfe6ad00" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 37499 } } peers { permanent_uuid: "057527a7327b4978babf4adcb767babb" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 38157 } } peers { permanent_uuid: "08ef2d53b3fe485586d1e76003807f15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 38473 } }
I20260929 16:46:37.465652 19555 leader_election.cc:288] T 5c20e4cdc2f84921a6e988e70b3a17ff P 08ef2d53b3fe485586d1e76003807f15 [CANDIDATE]: Term 1 election: Requested vote from peers 213bb71d8ab74a54ade6481abfe6ad00 (127.0.197.65:37499), 057527a7327b4978babf4adcb767babb (127.0.197.66:38157)
I20260929 16:46:37.466306 19216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c20e4cdc2f84921a6e988e70b3a17ff" candidate_uuid: "08ef2d53b3fe485586d1e76003807f15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "213bb71d8ab74a54ade6481abfe6ad00"
I20260929 16:46:37.466331 19350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c20e4cdc2f84921a6e988e70b3a17ff" candidate_uuid: "08ef2d53b3fe485586d1e76003807f15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "057527a7327b4978babf4adcb767babb"
I20260929 16:46:37.466555 19350 raft_consensus.cc:3037] T 5c20e4cdc2f84921a6e988e70b3a17ff P 057527a7327b4978babf4adcb767babb [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:46:37.466548 19216 raft_consensus.cc:3037] T 5c20e4cdc2f84921a6e988e70b3a17ff P 213bb71d8ab74a54ade6481abfe6ad00 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:46:37.469165 19350 raft_consensus.cc:2454] T 5c20e4cdc2f84921a6e988e70b3a17ff P 057527a7327b4978babf4adcb767babb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08ef2d53b3fe485586d1e76003807f15 in term 1.
I20260929 16:46:37.469275 19216 raft_consensus.cc:2454] T 5c20e4cdc2f84921a6e988e70b3a17ff P 213bb71d8ab74a54ade6481abfe6ad00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08ef2d53b3fe485586d1e76003807f15 in term 1.
I20260929 16:46:37.469743 19420 leader_election.cc:302] T 5c20e4cdc2f84921a6e988e70b3a17ff P 08ef2d53b3fe485586d1e76003807f15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08ef2d53b3fe485586d1e76003807f15, 213bb71d8ab74a54ade6481abfe6ad00; no voters:
I20260929 16:46:37.470094 19555 raft_consensus.cc:2781] T 5c20e4cdc2f84921a6e988e70b3a17ff P 08ef2d53b3fe485586d1e76003807f15 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:46:37.470624 19555 raft_consensus.cc:686] T 5c20e4cdc2f84921a6e988e70b3a17ff P 08ef2d53b3fe485586d1e76003807f15 [term 1 LEADER]: Becoming Leader. State: Replica: 08ef2d53b3fe485586d1e76003807f15, State: Running, Role: LEADER
I20260929 16:46:37.471256 19555 consensus_queue.cc:237] T 5c20e4cdc2f84921a6e988e70b3a17ff P 08ef2d53b3fe485586d1e76003807f15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "213bb71d8ab74a54ade6481abfe6ad00" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 37499 } } peers { permanent_uuid: "057527a7327b4978babf4adcb767babb" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 38157 } } peers { permanent_uuid: "08ef2d53b3fe485586d1e76003807f15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 38473 } }
I20260929 16:46:37.476377 19072 catalog_manager.cc:5495] T 5c20e4cdc2f84921a6e988e70b3a17ff P 08ef2d53b3fe485586d1e76003807f15 reported cstate change: term changed from 0 to 1, leader changed from <none> to 08ef2d53b3fe485586d1e76003807f15 (127.0.197.67). New cstate: current_term: 1 leader_uuid: "08ef2d53b3fe485586d1e76003807f15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "213bb71d8ab74a54ade6481abfe6ad00" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 37499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "057527a7327b4978babf4adcb767babb" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 38157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08ef2d53b3fe485586d1e76003807f15" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 38473 } health_report { overall_health: HEALTHY } } }
I20260929 16:46:37.524310 19545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:37.518336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40076 (local address 127.0.197.67:38473)
0929 16:46:37.518576 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:37.518581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:37.518606 (+ 25us) server_negotiation.cc:407] Connection header received
0929 16:46:37.518696 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:37.518701 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:37.518787 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:37.518919 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:46:37.519423 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:37.520141 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:37.521636 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:37.522080 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:37.522186 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:37.522836 (+ 650us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:46:37.523588 (+ 752us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:46:37.523729 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:37.523944 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 16:46:37.523990 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260929 16:46:37.533516 19350 raft_consensus.cc:1261] T 5c20e4cdc2f84921a6e988e70b3a17ff P 057527a7327b4978babf4adcb767babb [term 1 FOLLOWER]: Refusing update from remote peer 08ef2d53b3fe485586d1e76003807f15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:46:37.533566 19216 raft_consensus.cc:1261] T 5c20e4cdc2f84921a6e988e70b3a17ff P 213bb71d8ab74a54ade6481abfe6ad00 [term 1 FOLLOWER]: Refusing update from remote peer 08ef2d53b3fe485586d1e76003807f15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:46:37.534737 19558 consensus_queue.cc:1035] T 5c20e4cdc2f84921a6e988e70b3a17ff P 08ef2d53b3fe485586d1e76003807f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "057527a7327b4978babf4adcb767babb" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 38157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:46:37.535262 19555 consensus_queue.cc:1035] T 5c20e4cdc2f84921a6e988e70b3a17ff P 08ef2d53b3fe485586d1e76003807f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "213bb71d8ab74a54ade6481abfe6ad00" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 37499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:46:37.549015 19561 mvcc.cc:205] Tried to move back new op lower bound from 7334708828285554688 to 7334708828046704640. Current Snapshot: MvccSnapshot[applied={T|T < 7334708828285554688}]
I20260929 16:46:37.563220 19071 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:46:37.563541 19071 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:46:37.567780 19071 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7e63cd66df9f4909abd8f00da1419737: Sending DeleteTablet for 3 replicas of tablet 5c20e4cdc2f84921a6e988e70b3a17ff
I20260929 16:46:37.569403 19330 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c20e4cdc2f84921a6e988e70b3a17ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:46:37 UTC) from {username='kudu', principal='kudu/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:49568
I20260929 16:46:37.569410 19464 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c20e4cdc2f84921a6e988e70b3a17ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:46:37 UTC) from {username='kudu', principal='kudu/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:40072
I20260929 16:46:37.570327 19196 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c20e4cdc2f84921a6e988e70b3a17ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:46:37 UTC) from {username='kudu', principal='kudu/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:44262
I20260929 16:46:37.570510 19572 tablet_replica.cc:330] T 5c20e4cdc2f84921a6e988e70b3a17ff P 057527a7327b4978babf4adcb767babb: stopping tablet replica
I20260929 16:46:37.571152 19572 raft_consensus.cc:2229] T 5c20e4cdc2f84921a6e988e70b3a17ff P 057527a7327b4978babf4adcb767babb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:46:37.571774 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 19133
I20260929 16:46:37.571805 19572 raft_consensus.cc:2258] T 5c20e4cdc2f84921a6e988e70b3a17ff P 057527a7327b4978babf4adcb767babb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:46:37.572757 19571 tablet_replica.cc:330] T 5c20e4cdc2f84921a6e988e70b3a17ff P 08ef2d53b3fe485586d1e76003807f15: stopping tablet replica
I20260929 16:46:37.573293 19571 raft_consensus.cc:2229] T 5c20e4cdc2f84921a6e988e70b3a17ff P 08ef2d53b3fe485586d1e76003807f15 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:46:37.574076 19571 raft_consensus.cc:2258] T 5c20e4cdc2f84921a6e988e70b3a17ff P 08ef2d53b3fe485586d1e76003807f15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:46:37.576246 19571 ts_tablet_manager.cc:1905] T 5c20e4cdc2f84921a6e988e70b3a17ff P 08ef2d53b3fe485586d1e76003807f15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:46:37.584380 19571 ts_tablet_manager.cc:1918] T 5c20e4cdc2f84921a6e988e70b3a17ff P 08ef2d53b3fe485586d1e76003807f15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:46:37.584707 19571 log.cc:1196] T 5c20e4cdc2f84921a6e988e70b3a17ff P 08ef2d53b3fe485586d1e76003807f15: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/ts-2/wal/wals/5c20e4cdc2f84921a6e988e70b3a17ff
I20260929 16:46:37.585384 19571 ts_tablet_manager.cc:1939] T 5c20e4cdc2f84921a6e988e70b3a17ff P 08ef2d53b3fe485586d1e76003807f15: Deleting consensus metadata
I20260929 16:46:37.585335 19572 ts_tablet_manager.cc:1905] T 5c20e4cdc2f84921a6e988e70b3a17ff P 057527a7327b4978babf4adcb767babb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:46:37.587529 19056 catalog_manager.cc:4841] TS 08ef2d53b3fe485586d1e76003807f15 (127.0.197.67:38473): tablet 5c20e4cdc2f84921a6e988e70b3a17ff (table test-table [id=09d17e6d06de4c2a9becc00b217c7cb8]) successfully deleted
I20260929 16:46:37.595661 19573 tablet_replica.cc:330] T 5c20e4cdc2f84921a6e988e70b3a17ff P 213bb71d8ab74a54ade6481abfe6ad00: stopping tablet replica
I20260929 16:46:37.596197 19573 raft_consensus.cc:2229] T 5c20e4cdc2f84921a6e988e70b3a17ff P 213bb71d8ab74a54ade6481abfe6ad00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:46:37.596781 19573 raft_consensus.cc:2258] T 5c20e4cdc2f84921a6e988e70b3a17ff P 213bb71d8ab74a54ade6481abfe6ad00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:46:37.599409 19573 ts_tablet_manager.cc:1905] T 5c20e4cdc2f84921a6e988e70b3a17ff P 213bb71d8ab74a54ade6481abfe6ad00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:46:37.599910 19572 ts_tablet_manager.cc:1918] T 5c20e4cdc2f84921a6e988e70b3a17ff P 057527a7327b4978babf4adcb767babb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:46:37.600097 19572 log.cc:1196] T 5c20e4cdc2f84921a6e988e70b3a17ff P 057527a7327b4978babf4adcb767babb: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/ts-1/wal/wals/5c20e4cdc2f84921a6e988e70b3a17ff
I20260929 16:46:37.601955 19572 ts_tablet_manager.cc:1939] T 5c20e4cdc2f84921a6e988e70b3a17ff P 057527a7327b4978babf4adcb767babb: Deleting consensus metadata
Sep 29 16:46:37 dist-test-slave-l31f krb5kdc[19027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.65@KRBTEST.COM
I20260929 16:46:37.604101 19059 catalog_manager.cc:4841] TS 057527a7327b4978babf4adcb767babb (127.0.197.66:38157): tablet 5c20e4cdc2f84921a6e988e70b3a17ff (table test-table [id=09d17e6d06de4c2a9becc00b217c7cb8]) successfully deleted
I20260929 16:46:37.607539 19573 ts_tablet_manager.cc:1918] T 5c20e4cdc2f84921a6e988e70b3a17ff P 213bb71d8ab74a54ade6481abfe6ad00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:46:37.607784 19573 log.cc:1196] T 5c20e4cdc2f84921a6e988e70b3a17ff P 213bb71d8ab74a54ade6481abfe6ad00: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790700068255611-789-0/minicluster-data/ts-0/wal/wals/5c20e4cdc2f84921a6e988e70b3a17ff
I20260929 16:46:37.608328 19573 ts_tablet_manager.cc:1939] T 5c20e4cdc2f84921a6e988e70b3a17ff P 213bb71d8ab74a54ade6481abfe6ad00: Deleting consensus metadata
I20260929 16:46:37.610281 19059 catalog_manager.cc:4841] TS 213bb71d8ab74a54ade6481abfe6ad00 (127.0.197.65:37499): tablet 5c20e4cdc2f84921a6e988e70b3a17ff (table test-table [id=09d17e6d06de4c2a9becc00b217c7cb8]) successfully deleted
I20260929 16:46:37.612663 19543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:37.573556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44272 (local address 127.0.197.65:37499)
0929 16:46:37.573815 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:37.573826 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:37.591503 (+ 17677us) server_negotiation.cc:407] Connection header received
0929 16:46:37.592166 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:37.592175 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:37.592297 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:37.592393 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:37.594959 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:37.596104 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:37.597893 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:37.598537 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:37.598651 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:37.604625 (+ 5974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:37.604641 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:37.604661 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:46:37.604716 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:37.609067 (+ 4351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:37.610244 (+ 1177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:37.610255 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:37.610259 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:37.610461 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:37.610893 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:37.610899 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:37.610902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:37.611462 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:37.611805 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:37.612014 (+ 209us) server_negotiation.cc:299] Negotiation successful
0929 16:46:37.612259 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260929 16:46:37.613407 19256 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:46:37.707661 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 19133
I20260929 16:46:37.733260 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 19267
Sep 29 16:46:37 dist-test-slave-l31f krb5kdc[19027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.66@KRBTEST.COM
I20260929 16:46:37.751346 19547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:37.734147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49578 (local address 127.0.197.66:38157)
0929 16:46:37.734317 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:37.734322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:37.734488 (+ 166us) server_negotiation.cc:407] Connection header received
0929 16:46:37.734705 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:37.734711 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:37.734788 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:37.734901 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:37.736272 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:37.737011 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:37.738374 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:37.738886 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:37.739053 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:37.744001 (+ 4948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:37.744025 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:37.744048 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:46:37.744102 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:37.747959 (+ 3857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:37.748758 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:37.748768 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:37.748773 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:37.748940 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:37.749331 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:37.749337 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:37.749340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:37.749865 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:37.750171 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:37.750411 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 16:46:37.750662 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 16:46:37.752106 19390 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:46:37.844426 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 19267
I20260929 16:46:37.862803 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 19401
Sep 29 16:46:37 dist-test-slave-l31f krb5kdc[19027](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.67@KRBTEST.COM
I20260929 16:46:37.881088 19545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:37.863883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40088 (local address 127.0.197.67:38473)
0929 16:46:37.864115 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:37.864124 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:37.864145 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:46:37.864216 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:37.864220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:37.864284 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:37.864417 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:37.865779 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:37.866647 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:37.868199 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:37.868694 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:37.868865 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:37.874135 (+ 5270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:37.874149 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:37.874166 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:46:37.874223 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:37.877786 (+ 3563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:37.878652 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:37.878660 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:37.878663 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:37.878836 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:37.879315 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:37.879320 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:37.879330 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:37.879758 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:37.880117 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:37.880446 (+ 329us) server_negotiation.cc:299] Negotiation successful
0929 16:46:37.880711 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 16:46:37.881984 19524 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:46:38.012048 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 19401
I20260929 16:46:38.032481 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 19040
I20260929 16:46:38.033632 19102 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:46:38.146818 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 19040
2026-09-29T16:46:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4808 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-29T16:46:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:46:38Z Disabled control of system clock
I20260929 16:46:38.226651 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:43023
--webserver_interface=127.0.197.126
--webserver_port=0
--builtin_ntp_servers=127.0.197.84:45831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.126:43023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 16:46:38.580466 19582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:46:38.580809 19582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:46:38.580894 19582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:46:38.588966 19582 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:46:38.589115 19582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:46:38.589155 19582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:46:38.589219 19582 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:46:38.589265 19582 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:46:38.600075 19582 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:45831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.197.126:43023
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:43023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.197.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:46:38.601456 19582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:46:38.603545 19582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:46:38.616698 19587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:38.616823 19588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:38.616721 19590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:46:38.617586 19582 server_base.cc:1034] running on GCE node
I20260929 16:46:38.618470 19582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:46:38.620160 19582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:46:38.621395 19582 hybrid_clock.cc:648] HybridClock initialized: now 1790700398621330 us; error 60 us; skew 500 ppm
I20260929 16:46:38.621812 19582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:46:38.624446 19582 webserver.cc:460] Webserver started at http://127.0.197.126:34345/ using document root <none> and password file <none>
I20260929 16:46:38.625227 19582 fs_manager.cc:359] Metadata directory not provided
I20260929 16:46:38.625352 19582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:46:38.625705 19582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:46:38.628851 19582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/master-0/data/instance:
uuid: "57d7597ca39d4c9ea2472fd2ba2e54db"
format_stamp: "Formatted at 2026-09-29 16:46:38 on dist-test-slave-l31f"
server_key: "bfe7fbe75fc08adfeb39e614465160fc"
server_key_iv: "f1a02ffcc5a3ff59c9a3154e9efb5c31"
server_key_version: "encryptionkey@0"
I20260929 16:46:38.629750 19582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/master-0/wal/instance:
uuid: "57d7597ca39d4c9ea2472fd2ba2e54db"
format_stamp: "Formatted at 2026-09-29 16:46:38 on dist-test-slave-l31f"
server_key: "bfe7fbe75fc08adfeb39e614465160fc"
server_key_iv: "f1a02ffcc5a3ff59c9a3154e9efb5c31"
server_key_version: "encryptionkey@0"
I20260929 16:46:38.635486 19582 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 16:46:38.639763 19596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:38.641347 19582 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:46:38.641583 19582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "57d7597ca39d4c9ea2472fd2ba2e54db"
format_stamp: "Formatted at 2026-09-29 16:46:38 on dist-test-slave-l31f"
server_key: "bfe7fbe75fc08adfeb39e614465160fc"
server_key_iv: "f1a02ffcc5a3ff59c9a3154e9efb5c31"
server_key_version: "encryptionkey@0"
I20260929 16:46:38.641851 19582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:46:38.664192 19582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:46:38.665170 19582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:46:38.665488 19582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:46:38.686244 19582 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.126:43023
I20260929 16:46:38.686266 19647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.126:43023 every 8 connection(s)
I20260929 16:46:38.688078 19582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:46:38.692920 19648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:46:38.694332 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 19582
I20260929 16:46:38.694517 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/master-0/wal/instance
I20260929 16:46:38.694928 789 external_mini_cluster.cc:1428] Setting key 95cdd1cd75eaa0f5c113cc3e6c7b4ad6
I20260929 16:46:38.706544 19648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57d7597ca39d4c9ea2472fd2ba2e54db: Bootstrap starting.
I20260929 16:46:38.709044 19651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:38.696924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47932 (local address 127.0.197.126:43023)
0929 16:46:38.697973 (+ 1049us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:38.697986 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:38.698029 (+ 43us) server_negotiation.cc:407] Connection header received
0929 16:46:38.699916 (+ 1887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:38.699942 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:38.700492 (+ 550us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:38.700919 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:38.701516 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:38.702788 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:38.704228 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:38.704973 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:38.705191 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:38.706470 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:38.706478 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:38.706494 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:46:38.707204 (+ 710us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:38.707298 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:46:38.707321 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:38.707463 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:38.707708 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 16:46:38.708031 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":337,"threads_started":1}
I20260929 16:46:38.710467 19648 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57d7597ca39d4c9ea2472fd2ba2e54db: Neither blocks nor log segments found. Creating new log.
I20260929 16:46:38.711735 19648 log.cc:824] T 00000000000000000000000000000000 P 57d7597ca39d4c9ea2472fd2ba2e54db: Log is configured to *not* fsync() on all Append() calls
I20260929 16:46:38.714795 19648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57d7597ca39d4c9ea2472fd2ba2e54db: No bootstrap required, opened a new log
I20260929 16:46:38.723145 19648 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57d7597ca39d4c9ea2472fd2ba2e54db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57d7597ca39d4c9ea2472fd2ba2e54db" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 43023 } }
I20260929 16:46:38.723595 19648 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57d7597ca39d4c9ea2472fd2ba2e54db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:46:38.723699 19648 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57d7597ca39d4c9ea2472fd2ba2e54db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57d7597ca39d4c9ea2472fd2ba2e54db, State: Initialized, Role: FOLLOWER
I20260929 16:46:38.724363 19648 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57d7597ca39d4c9ea2472fd2ba2e54db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57d7597ca39d4c9ea2472fd2ba2e54db" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 43023 } }
I20260929 16:46:38.724637 19648 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57d7597ca39d4c9ea2472fd2ba2e54db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:46:38.724788 19648 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57d7597ca39d4c9ea2472fd2ba2e54db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:46:38.724975 19648 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57d7597ca39d4c9ea2472fd2ba2e54db [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:46:38.727674 19648 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57d7597ca39d4c9ea2472fd2ba2e54db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57d7597ca39d4c9ea2472fd2ba2e54db" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 43023 } }
I20260929 16:46:38.728237 19648 leader_election.cc:302] T 00000000000000000000000000000000 P 57d7597ca39d4c9ea2472fd2ba2e54db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57d7597ca39d4c9ea2472fd2ba2e54db; no voters:
I20260929 16:46:38.728946 19648 leader_election.cc:288] T 00000000000000000000000000000000 P 57d7597ca39d4c9ea2472fd2ba2e54db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:46:38.729055 19653 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57d7597ca39d4c9ea2472fd2ba2e54db [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:46:38.729951 19653 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57d7597ca39d4c9ea2472fd2ba2e54db [term 1 LEADER]: Becoming Leader. State: Replica: 57d7597ca39d4c9ea2472fd2ba2e54db, State: Running, Role: LEADER
I20260929 16:46:38.730497 19653 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57d7597ca39d4c9ea2472fd2ba2e54db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57d7597ca39d4c9ea2472fd2ba2e54db" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 43023 } }
I20260929 16:46:38.731393 19648 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57d7597ca39d4c9ea2472fd2ba2e54db [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:46:38.733615 19655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57d7597ca39d4c9ea2472fd2ba2e54db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57d7597ca39d4c9ea2472fd2ba2e54db. Latest consensus state: current_term: 1 leader_uuid: "57d7597ca39d4c9ea2472fd2ba2e54db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57d7597ca39d4c9ea2472fd2ba2e54db" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 43023 } } }
I20260929 16:46:38.734884 19655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57d7597ca39d4c9ea2472fd2ba2e54db [sys.catalog]: This master's current role is: LEADER
I20260929 16:46:38.737469 19661 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:46:38.737325 19654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57d7597ca39d4c9ea2472fd2ba2e54db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57d7597ca39d4c9ea2472fd2ba2e54db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57d7597ca39d4c9ea2472fd2ba2e54db" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 43023 } } }
I20260929 16:46:38.737663 19654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57d7597ca39d4c9ea2472fd2ba2e54db [sys.catalog]: This master's current role is: LEADER
I20260929 16:46:38.742867 19661 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:46:38.753608 19661 catalog_manager.cc:1347] Generated new cluster ID: 83572ef0f81b4c5bb28dcd7377229aaa
I20260929 16:46:38.753834 19661 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:46:38.765769 19661 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:46:38.767768 19661 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:46:38.781718 19661 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57d7597ca39d4c9ea2472fd2ba2e54db: Generated new TSK 0
I20260929 16:46:38.782749 19661 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:46:38.796281 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.65:0
--local_ip_for_outbound_sockets=127.0.197.65
--webserver_interface=127.0.197.65
--webserver_port=0
--tserver_master_addrs=127.0.197.126:43023
--builtin_ntp_servers=127.0.197.84:45831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 16:46:39.166045 19672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:46:39.166368 19672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:46:39.166523 19672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:46:39.174404 19672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:46:39.174535 19672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:46:39.174702 19672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.65
I20260929 16:46:39.184537 19672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:45831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.197.65
--webserver_port=0
--tserver_master_addrs=127.0.197.126:43023
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.65
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:46:39.185936 19672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:46:39.188027 19672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:46:39.202349 19678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:39.202302 19677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:39.202374 19680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:46:39.203773 19672 server_base.cc:1034] running on GCE node
I20260929 16:46:39.204495 19672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:46:39.205598 19672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:46:39.206863 19672 hybrid_clock.cc:648] HybridClock initialized: now 1790700399206806 us; error 73 us; skew 500 ppm
I20260929 16:46:39.207278 19672 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:46:39.209925 19672 webserver.cc:460] Webserver started at http://127.0.197.65:33827/ using document root <none> and password file <none>
I20260929 16:46:39.210739 19672 fs_manager.cc:359] Metadata directory not provided
I20260929 16:46:39.210896 19672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:46:39.211316 19672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:46:39.214545 19672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/ts-0/data/instance:
uuid: "8f95a162b7f34cffbc5a311371ad49ca"
format_stamp: "Formatted at 2026-09-29 16:46:39 on dist-test-slave-l31f"
server_key: "00f40fddc158e4aede93ab40c3107fcb"
server_key_iv: "eef45ec2323fe5d3f6fb4aaff9e4fa7b"
server_key_version: "encryptionkey@0"
I20260929 16:46:39.215512 19672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/ts-0/wal/instance:
uuid: "8f95a162b7f34cffbc5a311371ad49ca"
format_stamp: "Formatted at 2026-09-29 16:46:39 on dist-test-slave-l31f"
server_key: "00f40fddc158e4aede93ab40c3107fcb"
server_key_iv: "eef45ec2323fe5d3f6fb4aaff9e4fa7b"
server_key_version: "encryptionkey@0"
I20260929 16:46:39.221450 19672 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 16:46:39.225685 19686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:39.227396 19672 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:46:39.227622 19672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/ts-0/wal
uuid: "8f95a162b7f34cffbc5a311371ad49ca"
format_stamp: "Formatted at 2026-09-29 16:46:39 on dist-test-slave-l31f"
server_key: "00f40fddc158e4aede93ab40c3107fcb"
server_key_iv: "eef45ec2323fe5d3f6fb4aaff9e4fa7b"
server_key_version: "encryptionkey@0"
I20260929 16:46:39.227893 19672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:46:39.252197 19672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:46:39.253154 19672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:46:39.253508 19672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:46:39.254753 19672 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:46:39.256587 19672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:46:39.256738 19672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:39.256878 19672 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:46:39.256968 19672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:39.293118 19672 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.65:40017
I20260929 16:46:39.293176 19798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.65:40017 every 8 connection(s)
I20260929 16:46:39.294590 19672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
I20260929 16:46:39.304229 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 19672
I20260929 16:46:39.304438 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/ts-0/wal/instance
I20260929 16:46:39.304888 789 external_mini_cluster.cc:1428] Setting key 2ade25f7eb72ce84f4b9816ae93a55e1
I20260929 16:46:39.308061 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.66:0
--local_ip_for_outbound_sockets=127.0.197.66
--webserver_interface=127.0.197.66
--webserver_port=0
--tserver_master_addrs=127.0.197.126:43023
--builtin_ntp_servers=127.0.197.84:45831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 16:46:39.308844 19801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:39.297533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:51293 (local address 127.0.197.126:43023)
0929 16:46:39.298050 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:39.298057 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:39.299407 (+ 1350us) server_negotiation.cc:407] Connection header received
0929 16:46:39.301296 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:39.301300 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:39.301380 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:39.301487 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:39.302692 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:39.303586 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:39.305508 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:39.305971 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:39.306080 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:39.307902 (+ 1822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:39.307907 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:39.307911 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:46:39.307930 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:39.307971 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:46:39.307984 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:39.308093 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:39.308369 (+ 276us) server_negotiation.cc:299] Negotiation successful
0929 16:46:39.308481 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":251,"threads_started":1}
I20260929 16:46:39.309954 19802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:39.298185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:43023 (local address 127.0.197.65:51293)
0929 16:46:39.299233 (+ 1048us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:39.299278 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 16:46:39.300981 (+ 1703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:46:39.301720 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:46:39.301729 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:46:39.301828 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:46:39.302494 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:39.302511 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:39.303758 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:39.303762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:39.305315 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:39.305329 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:39.306230 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:39.306237 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:39.306565 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:39.307520 (+ 955us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:46:39.307523 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:46:39.307567 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:46:39.307659 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:46:39.308106 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:46:39.308120 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:46:39.308135 (+ 15us) client_negotiation.cc:769] Sending connection context
0929 16:46:39.308344 (+ 209us) client_negotiation.cc:240] Negotiation successful
0929 16:46:39.308788 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":323,"threads_started":1}
I20260929 16:46:39.311803 19799 heartbeater.cc:347] Connected to a master server at 127.0.197.126:43023
I20260929 16:46:39.312213 19799 heartbeater.cc:464] Registering TS with master...
I20260929 16:46:39.313082 19799 heartbeater.cc:511] Master 127.0.197.126:43023 requested a full tablet report, sending...
I20260929 16:46:39.315759 19613 ts_manager.cc:194] Registered new tserver with Master: 8f95a162b7f34cffbc5a311371ad49ca (127.0.197.65:40017)
I20260929 16:46:39.317404 19613 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.197.65:51293
W20260929 16:46:39.664428 19803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:46:39.664767 19803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:46:39.664857 19803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:46:39.672674 19803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:46:39.672806 19803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:46:39.672919 19803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.66
I20260929 16:46:39.682755 19803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:45831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.197.66
--webserver_port=0
--tserver_master_addrs=127.0.197.126:43023
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.66
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:46:39.684113 19803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:46:39.686208 19803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:46:39.703088 19808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:39.703217 19809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:39.703595 19811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:46:39.703884 19803 server_base.cc:1034] running on GCE node
I20260929 16:46:39.704564 19803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:46:39.705667 19803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:46:39.706925 19803 hybrid_clock.cc:648] HybridClock initialized: now 1790700399706858 us; error 60 us; skew 500 ppm
I20260929 16:46:39.707314 19803 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:46:39.709693 19803 webserver.cc:460] Webserver started at http://127.0.197.66:43991/ using document root <none> and password file <none>
I20260929 16:46:39.710444 19803 fs_manager.cc:359] Metadata directory not provided
I20260929 16:46:39.710595 19803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:46:39.710938 19803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:46:39.714002 19803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/ts-1/data/instance:
uuid: "fd2272aee3234376a621b601a5739b44"
format_stamp: "Formatted at 2026-09-29 16:46:39 on dist-test-slave-l31f"
server_key: "ec3b6149196c6c1646aa1931d2597a05"
server_key_iv: "369546cdd4d069c25f7e792d3a5e9292"
server_key_version: "encryptionkey@0"
I20260929 16:46:39.714901 19803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/ts-1/wal/instance:
uuid: "fd2272aee3234376a621b601a5739b44"
format_stamp: "Formatted at 2026-09-29 16:46:39 on dist-test-slave-l31f"
server_key: "ec3b6149196c6c1646aa1931d2597a05"
server_key_iv: "369546cdd4d069c25f7e792d3a5e9292"
server_key_version: "encryptionkey@0"
I20260929 16:46:39.720705 19803 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260929 16:46:39.724975 19817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:39.726770 19803 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260929 16:46:39.727119 19803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/ts-1/wal
uuid: "fd2272aee3234376a621b601a5739b44"
format_stamp: "Formatted at 2026-09-29 16:46:39 on dist-test-slave-l31f"
server_key: "ec3b6149196c6c1646aa1931d2597a05"
server_key_iv: "369546cdd4d069c25f7e792d3a5e9292"
server_key_version: "encryptionkey@0"
I20260929 16:46:39.727391 19803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:46:39.747642 19803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:46:39.748605 19803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:46:39.748950 19803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:46:39.749989 19803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:46:39.751693 19803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:46:39.751823 19803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:39.751959 19803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:46:39.752050 19803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:39.787838 19803 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.66:42299
I20260929 16:46:39.787952 19929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.66:42299 every 8 connection(s)
I20260929 16:46:39.789395 19803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
I20260929 16:46:39.798527 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 19803
I20260929 16:46:39.798712 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/ts-1/wal/instance
I20260929 16:46:39.799291 789 external_mini_cluster.cc:1428] Setting key c6114b633346463c6c80331bf873502f
I20260929 16:46:39.802251 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.67:0
--local_ip_for_outbound_sockets=127.0.197.67
--webserver_interface=127.0.197.67
--webserver_port=0
--tserver_master_addrs=127.0.197.126:43023
--builtin_ntp_servers=127.0.197.84:45831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 16:46:39.804080 19801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:39.792367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:43541 (local address 127.0.197.126:43023)
0929 16:46:39.792554 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:39.792559 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:39.794129 (+ 1570us) server_negotiation.cc:407] Connection header received
0929 16:46:39.796062 (+ 1933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:39.796067 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:39.796151 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:39.796248 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:39.797857 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:39.798952 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:39.800946 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:39.801359 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:39.801518 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:39.803197 (+ 1679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:39.803203 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:39.803206 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:46:39.803228 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:39.803263 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:46:39.803277 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:39.803375 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:39.803634 (+ 259us) server_negotiation.cc:299] Negotiation successful
0929 16:46:39.803750 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 16:46:39.805279 19932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:39.792919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:43023 (local address 127.0.197.66:43541)
0929 16:46:39.793916 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:39.793973 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 16:46:39.795780 (+ 1807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:46:39.796481 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:46:39.796492 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:46:39.796595 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:46:39.797656 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:39.797677 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:39.799183 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:39.799187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:39.800762 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:39.800776 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:39.801614 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:39.801622 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:39.801877 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:39.802711 (+ 834us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:46:39.802714 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:46:39.802761 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:46:39.802859 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:46:39.803392 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:46:39.803406 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:46:39.803415 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 16:46:39.803624 (+ 209us) client_negotiation.cc:240] Negotiation successful
0929 16:46:39.804118 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":319,"threads_started":1}
I20260929 16:46:39.807137 19930 heartbeater.cc:347] Connected to a master server at 127.0.197.126:43023
I20260929 16:46:39.807507 19930 heartbeater.cc:464] Registering TS with master...
I20260929 16:46:39.808384 19930 heartbeater.cc:511] Master 127.0.197.126:43023 requested a full tablet report, sending...
I20260929 16:46:39.810184 19613 ts_manager.cc:194] Registered new tserver with Master: fd2272aee3234376a621b601a5739b44 (127.0.197.66:42299)
I20260929 16:46:39.811187 19613 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.197.66:43541
W20260929 16:46:40.150377 19933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:46:40.150708 19933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:46:40.150804 19933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:46:40.158377 19933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:46:40.158535 19933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:46:40.158655 19933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.67
I20260929 16:46:40.168371 19933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:45831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.197.67
--webserver_port=0
--tserver_master_addrs=127.0.197.126:43023
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.67
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:46:40.169812 19933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:46:40.171976 19933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:46:40.188201 19941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:40.188262 19938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:40.188295 19939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:46:40.189419 19933 server_base.cc:1034] running on GCE node
I20260929 16:46:40.190354 19933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:46:40.191624 19933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:46:40.192919 19933 hybrid_clock.cc:648] HybridClock initialized: now 1790700400192873 us; error 78 us; skew 500 ppm
I20260929 16:46:40.193316 19933 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:46:40.196081 19933 webserver.cc:460] Webserver started at http://127.0.197.67:41207/ using document root <none> and password file <none>
I20260929 16:46:40.196980 19933 fs_manager.cc:359] Metadata directory not provided
I20260929 16:46:40.197129 19933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:46:40.197484 19933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:46:40.200834 19933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/ts-2/data/instance:
uuid: "2e788361fbd54b6590fdb0efdc4c57b4"
format_stamp: "Formatted at 2026-09-29 16:46:40 on dist-test-slave-l31f"
server_key: "65883522d211347bd38ac0277f42e821"
server_key_iv: "b25ae76832d7bf4c6b97ec29706c7a63"
server_key_version: "encryptionkey@0"
I20260929 16:46:40.201768 19933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/ts-2/wal/instance:
uuid: "2e788361fbd54b6590fdb0efdc4c57b4"
format_stamp: "Formatted at 2026-09-29 16:46:40 on dist-test-slave-l31f"
server_key: "65883522d211347bd38ac0277f42e821"
server_key_iv: "b25ae76832d7bf4c6b97ec29706c7a63"
server_key_version: "encryptionkey@0"
I20260929 16:46:40.207990 19933 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260929 16:46:40.212128 19947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:40.213685 19933 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:46:40.213996 19933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/ts-2/wal
uuid: "2e788361fbd54b6590fdb0efdc4c57b4"
format_stamp: "Formatted at 2026-09-29 16:46:40 on dist-test-slave-l31f"
server_key: "65883522d211347bd38ac0277f42e821"
server_key_iv: "b25ae76832d7bf4c6b97ec29706c7a63"
server_key_version: "encryptionkey@0"
I20260929 16:46:40.214267 19933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:46:40.238039 19933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:46:40.238994 19933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:46:40.239418 19933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:46:40.240563 19933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:46:40.242273 19933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:46:40.242408 19933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:40.242545 19933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:46:40.242636 19933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:40.278857 19933 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.67:43897
I20260929 16:46:40.278960 20059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.67:43897 every 8 connection(s)
I20260929 16:46:40.280277 19933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
I20260929 16:46:40.281260 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 19933
I20260929 16:46:40.281453 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/ts-2/wal/instance
I20260929 16:46:40.281983 789 external_mini_cluster.cc:1428] Setting key 4fa21f08f83b1e51f9a0ea0d5568c20b
I20260929 16:46:40.294655 19801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:40.283344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:39615 (local address 127.0.197.126:43023)
0929 16:46:40.283571 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:40.283577 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:40.285532 (+ 1955us) server_negotiation.cc:407] Connection header received
0929 16:46:40.287351 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:40.287356 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:40.287434 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:40.287524 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:40.288702 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:40.289629 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:40.291492 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:40.291921 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:40.292038 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:40.293756 (+ 1718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:40.293762 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:40.293765 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:46:40.293787 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:40.293822 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:46:40.293835 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:40.293934 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:40.294192 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 16:46:40.294310 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 16:46:40.295831 20062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:40.284018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:43023 (local address 127.0.197.67:39615)
0929 16:46:40.285238 (+ 1220us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:40.285294 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 16:46:40.286974 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:46:40.287727 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:46:40.287735 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:46:40.287840 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:46:40.288501 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:40.288522 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:40.289843 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:40.289856 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:40.291284 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:40.291299 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:40.292176 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:40.292182 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:40.292452 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:40.293358 (+ 906us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:46:40.293361 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:46:40.293411 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:46:40.293504 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:46:40.293952 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:46:40.293973 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:46:40.293980 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 16:46:40.294180 (+ 200us) client_negotiation.cc:240] Negotiation successful
0929 16:46:40.294645 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":321,"threads_started":1}
I20260929 16:46:40.297910 20060 heartbeater.cc:347] Connected to a master server at 127.0.197.126:43023
I20260929 16:46:40.298290 20060 heartbeater.cc:464] Registering TS with master...
I20260929 16:46:40.299214 20060 heartbeater.cc:511] Master 127.0.197.126:43023 requested a full tablet report, sending...
I20260929 16:46:40.300696 19613 ts_manager.cc:194] Registered new tserver with Master: 2e788361fbd54b6590fdb0efdc4c57b4 (127.0.197.67:43897)
I20260929 16:46:40.301571 19613 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.197.67:39615
I20260929 16:46:40.311246 789 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:46:40.321619 19799 heartbeater.cc:503] Master 127.0.197.126:43023 was elected leader, sending a full tablet report...
I20260929 16:46:40.323432 19801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:40.316114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47940 (local address 127.0.197.126:43023)
0929 16:46:40.316306 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:40.316311 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:40.316797 (+ 486us) server_negotiation.cc:407] Connection header received
0929 16:46:40.316941 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:40.316946 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:40.317017 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:40.317114 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:40.317714 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:40.318540 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:40.320179 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:40.320628 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:40.320737 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:40.322154 (+ 1417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:40.322163 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:40.322171 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:46:40.322199 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:40.322365 (+ 166us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:46:40.322413 (+ 48us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:40.322622 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:40.322957 (+ 335us) server_negotiation.cc:299] Negotiation successful
0929 16:46:40.323060 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 16:46:40.328249 19613 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47940:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:46:40.336328 19613 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:46:40.374085 20072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:40.349622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.67:43897 (local address 127.0.0.1:55224)
0929 16:46:40.362457 (+ 12835us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:40.362485 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 16:46:40.362688 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:46:40.363671 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:46:40.363680 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:46:40.363727 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:46:40.364392 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:40.364411 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:40.366792 (+ 2381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:40.366801 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:40.370412 (+ 3611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:40.370438 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:40.372958 (+ 2520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:40.372965 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:40.373213 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:40.373276 (+ 63us) client_negotiation.cc:769] Sending connection context
0929 16:46:40.373442 (+ 166us) client_negotiation.cc:240] Negotiation successful
0929 16:46:40.373666 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12646,"mutex_wait_us":2,"thread_start_us":698,"threads_started":1}
I20260929 16:46:40.374089 20075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:40.350003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55224 (local address 127.0.197.67:43897)
0929 16:46:40.361692 (+ 11689us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:40.361706 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:40.362862 (+ 1156us) server_negotiation.cc:407] Connection header received
0929 16:46:40.363055 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:40.363067 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:40.363373 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:40.364680 (+ 1307us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:46:40.365092 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:40.366583 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:40.371098 (+ 4515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:40.372729 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:40.372886 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:40.373178 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:40.373524 (+ 346us) server_negotiation.cc:299] Negotiation successful
0929 16:46:40.373715 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11460,"thread_start_us":11284,"threads_started":1}
I20260929 16:46:40.374788 20074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:40.349761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38310 (local address 127.0.197.65:40017)
0929 16:46:40.362091 (+ 12330us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:40.362102 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:40.362148 (+ 46us) server_negotiation.cc:407] Connection header received
0929 16:46:40.362295 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:40.362308 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:40.362728 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:40.363094 (+ 366us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:46:40.364619 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:40.366902 (+ 2283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:40.371493 (+ 4591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:40.372684 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:40.372780 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:40.373129 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:40.373948 (+ 819us) server_negotiation.cc:299] Negotiation successful
0929 16:46:40.374147 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12095,"thread_start_us":11936,"threads_started":1}
I20260929 16:46:40.375401 20073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:40.349499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:40017 (local address 127.0.0.1:38310)
0929 16:46:40.350380 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:40.350423 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 16:46:40.350633 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:46:40.363501 (+ 12868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:46:40.363514 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:46:40.363564 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:46:40.364269 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:40.364293 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:40.367162 (+ 2869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:40.367171 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:40.371235 (+ 4064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:40.371250 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:40.373316 (+ 2066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:40.373322 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:40.373508 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:40.373537 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 16:46:40.373611 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 16:46:40.373721 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"mutex_wait_us":1075,"thread_start_us":12577,"threads_started":1}
I20260929 16:46:40.379406 20076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:40.351845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.66:42299 (local address 127.0.0.1:42568)
0929 16:46:40.370999 (+ 19154us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:40.371046 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 16:46:40.371199 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:46:40.372294 (+ 1095us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:46:40.372298 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:46:40.372328 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:46:40.372689 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:40.372702 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:40.374593 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:40.374602 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:40.376270 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:40.376285 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:40.378574 (+ 2289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:40.378581 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:40.378724 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:40.378750 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 16:46:40.378869 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 16:46:40.379070 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18963,"thread_start_us":18793,"threads_started":1}
I20260929 16:46:40.379453 20077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:40.352115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42568 (local address 127.0.197.66:42299)
0929 16:46:40.370400 (+ 18285us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:40.370412 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:40.371341 (+ 929us) server_negotiation.cc:407] Connection header received
0929 16:46:40.371472 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:40.371481 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:40.371789 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:40.372042 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:46:40.372950 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:40.374449 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:40.376470 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:40.378017 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:40.378170 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:40.378424 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:40.378935 (+ 511us) server_negotiation.cc:299] Negotiation successful
0929 16:46:40.379134 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17901,"thread_start_us":17736,"threads_started":1}
I20260929 16:46:40.379071 19995 tablet_service.cc:1467] Processing CreateTablet for tablet bb24460cfd5844dcaf821309598ef484 (DEFAULT_TABLE table=test-table [id=4b3f22eb6a5f4259b8121be228235ca4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:46:40.379071 19734 tablet_service.cc:1467] Processing CreateTablet for tablet bb24460cfd5844dcaf821309598ef484 (DEFAULT_TABLE table=test-table [id=4b3f22eb6a5f4259b8121be228235ca4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:46:40.380752 19734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb24460cfd5844dcaf821309598ef484. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:46:40.380765 19995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb24460cfd5844dcaf821309598ef484. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:46:40.383826 19865 tablet_service.cc:1467] Processing CreateTablet for tablet bb24460cfd5844dcaf821309598ef484 (DEFAULT_TABLE table=test-table [id=4b3f22eb6a5f4259b8121be228235ca4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:46:40.385365 19865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb24460cfd5844dcaf821309598ef484. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:46:40.397233 20078 tablet_bootstrap.cc:492] T bb24460cfd5844dcaf821309598ef484 P 8f95a162b7f34cffbc5a311371ad49ca: Bootstrap starting.
I20260929 16:46:40.398272 20079 tablet_bootstrap.cc:492] T bb24460cfd5844dcaf821309598ef484 P 2e788361fbd54b6590fdb0efdc4c57b4: Bootstrap starting.
I20260929 16:46:40.401948 20079 tablet_bootstrap.cc:654] T bb24460cfd5844dcaf821309598ef484 P 2e788361fbd54b6590fdb0efdc4c57b4: Neither blocks nor log segments found. Creating new log.
I20260929 16:46:40.402983 20078 tablet_bootstrap.cc:654] T bb24460cfd5844dcaf821309598ef484 P 8f95a162b7f34cffbc5a311371ad49ca: Neither blocks nor log segments found. Creating new log.
I20260929 16:46:40.403059 20080 tablet_bootstrap.cc:492] T bb24460cfd5844dcaf821309598ef484 P fd2272aee3234376a621b601a5739b44: Bootstrap starting.
I20260929 16:46:40.403512 20079 log.cc:824] T bb24460cfd5844dcaf821309598ef484 P 2e788361fbd54b6590fdb0efdc4c57b4: Log is configured to *not* fsync() on all Append() calls
I20260929 16:46:40.404343 20078 log.cc:824] T bb24460cfd5844dcaf821309598ef484 P 8f95a162b7f34cffbc5a311371ad49ca: Log is configured to *not* fsync() on all Append() calls
I20260929 16:46:40.407248 20080 tablet_bootstrap.cc:654] T bb24460cfd5844dcaf821309598ef484 P fd2272aee3234376a621b601a5739b44: Neither blocks nor log segments found. Creating new log.
I20260929 16:46:40.407461 20079 tablet_bootstrap.cc:492] T bb24460cfd5844dcaf821309598ef484 P 2e788361fbd54b6590fdb0efdc4c57b4: No bootstrap required, opened a new log
I20260929 16:46:40.407778 20079 ts_tablet_manager.cc:1397] T bb24460cfd5844dcaf821309598ef484 P 2e788361fbd54b6590fdb0efdc4c57b4: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260929 16:46:40.408473 20080 log.cc:824] T bb24460cfd5844dcaf821309598ef484 P fd2272aee3234376a621b601a5739b44: Log is configured to *not* fsync() on all Append() calls
I20260929 16:46:40.409451 20078 tablet_bootstrap.cc:492] T bb24460cfd5844dcaf821309598ef484 P 8f95a162b7f34cffbc5a311371ad49ca: No bootstrap required, opened a new log
I20260929 16:46:40.409757 20078 ts_tablet_manager.cc:1397] T bb24460cfd5844dcaf821309598ef484 P 8f95a162b7f34cffbc5a311371ad49ca: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.009s
I20260929 16:46:40.411477 20080 tablet_bootstrap.cc:492] T bb24460cfd5844dcaf821309598ef484 P fd2272aee3234376a621b601a5739b44: No bootstrap required, opened a new log
I20260929 16:46:40.411788 20080 ts_tablet_manager.cc:1397] T bb24460cfd5844dcaf821309598ef484 P fd2272aee3234376a621b601a5739b44: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260929 16:46:40.419288 20079 raft_consensus.cc:348] T bb24460cfd5844dcaf821309598ef484 P 2e788361fbd54b6590fdb0efdc4c57b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f95a162b7f34cffbc5a311371ad49ca" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 40017 } } peers { permanent_uuid: "2e788361fbd54b6590fdb0efdc4c57b4" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 43897 } } peers { permanent_uuid: "fd2272aee3234376a621b601a5739b44" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42299 } }
I20260929 16:46:40.419819 20079 raft_consensus.cc:374] T bb24460cfd5844dcaf821309598ef484 P 2e788361fbd54b6590fdb0efdc4c57b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:46:40.419917 20079 raft_consensus.cc:729] T bb24460cfd5844dcaf821309598ef484 P 2e788361fbd54b6590fdb0efdc4c57b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e788361fbd54b6590fdb0efdc4c57b4, State: Initialized, Role: FOLLOWER
I20260929 16:46:40.420277 20080 raft_consensus.cc:348] T bb24460cfd5844dcaf821309598ef484 P fd2272aee3234376a621b601a5739b44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f95a162b7f34cffbc5a311371ad49ca" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 40017 } } peers { permanent_uuid: "2e788361fbd54b6590fdb0efdc4c57b4" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 43897 } } peers { permanent_uuid: "fd2272aee3234376a621b601a5739b44" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42299 } }
I20260929 16:46:40.420718 20080 raft_consensus.cc:374] T bb24460cfd5844dcaf821309598ef484 P fd2272aee3234376a621b601a5739b44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:46:40.420677 20079 consensus_queue.cc:260] T bb24460cfd5844dcaf821309598ef484 P 2e788361fbd54b6590fdb0efdc4c57b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f95a162b7f34cffbc5a311371ad49ca" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 40017 } } peers { permanent_uuid: "2e788361fbd54b6590fdb0efdc4c57b4" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 43897 } } peers { permanent_uuid: "fd2272aee3234376a621b601a5739b44" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42299 } }
I20260929 16:46:40.420848 20080 raft_consensus.cc:729] T bb24460cfd5844dcaf821309598ef484 P fd2272aee3234376a621b601a5739b44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd2272aee3234376a621b601a5739b44, State: Initialized, Role: FOLLOWER
I20260929 16:46:40.420753 20078 raft_consensus.cc:348] T bb24460cfd5844dcaf821309598ef484 P 8f95a162b7f34cffbc5a311371ad49ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f95a162b7f34cffbc5a311371ad49ca" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 40017 } } peers { permanent_uuid: "2e788361fbd54b6590fdb0efdc4c57b4" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 43897 } } peers { permanent_uuid: "fd2272aee3234376a621b601a5739b44" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42299 } }
I20260929 16:46:40.421443 20080 consensus_queue.cc:260] T bb24460cfd5844dcaf821309598ef484 P fd2272aee3234376a621b601a5739b44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f95a162b7f34cffbc5a311371ad49ca" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 40017 } } peers { permanent_uuid: "2e788361fbd54b6590fdb0efdc4c57b4" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 43897 } } peers { permanent_uuid: "fd2272aee3234376a621b601a5739b44" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42299 } }
I20260929 16:46:40.421660 20078 raft_consensus.cc:374] T bb24460cfd5844dcaf821309598ef484 P 8f95a162b7f34cffbc5a311371ad49ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:46:40.421795 20078 raft_consensus.cc:729] T bb24460cfd5844dcaf821309598ef484 P 8f95a162b7f34cffbc5a311371ad49ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f95a162b7f34cffbc5a311371ad49ca, State: Initialized, Role: FOLLOWER
I20260929 16:46:40.422521 20078 consensus_queue.cc:260] T bb24460cfd5844dcaf821309598ef484 P 8f95a162b7f34cffbc5a311371ad49ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f95a162b7f34cffbc5a311371ad49ca" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 40017 } } peers { permanent_uuid: "2e788361fbd54b6590fdb0efdc4c57b4" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 43897 } } peers { permanent_uuid: "fd2272aee3234376a621b601a5739b44" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42299 } }
I20260929 16:46:40.426980 20078 ts_tablet_manager.cc:1428] T bb24460cfd5844dcaf821309598ef484 P 8f95a162b7f34cffbc5a311371ad49ca: Time spent starting tablet: real 0.017s user 0.012s sys 0.005s
I20260929 16:46:40.428225 20060 heartbeater.cc:503] Master 127.0.197.126:43023 was elected leader, sending a full tablet report...
I20260929 16:46:40.428687 19930 heartbeater.cc:503] Master 127.0.197.126:43023 was elected leader, sending a full tablet report...
I20260929 16:46:40.429001 20079 ts_tablet_manager.cc:1428] T bb24460cfd5844dcaf821309598ef484 P 2e788361fbd54b6590fdb0efdc4c57b4: Time spent starting tablet: real 0.021s user 0.013s sys 0.008s
I20260929 16:46:40.430867 20080 ts_tablet_manager.cc:1428] T bb24460cfd5844dcaf821309598ef484 P fd2272aee3234376a621b601a5739b44: Time spent starting tablet: real 0.019s user 0.014s sys 0.001s
W20260929 16:46:40.532155 20061 tablet.cc:2369] T bb24460cfd5844dcaf821309598ef484 P 2e788361fbd54b6590fdb0efdc4c57b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:46:40.541966 19931 tablet.cc:2369] T bb24460cfd5844dcaf821309598ef484 P fd2272aee3234376a621b601a5739b44: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:46:40.547336 19800 tablet.cc:2369] T bb24460cfd5844dcaf821309598ef484 P 8f95a162b7f34cffbc5a311371ad49ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:46:40.567548 20086 raft_consensus.cc:482] T bb24460cfd5844dcaf821309598ef484 P 8f95a162b7f34cffbc5a311371ad49ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:46:40.567929 20086 raft_consensus.cc:504] T bb24460cfd5844dcaf821309598ef484 P 8f95a162b7f34cffbc5a311371ad49ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f95a162b7f34cffbc5a311371ad49ca" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 40017 } } peers { permanent_uuid: "2e788361fbd54b6590fdb0efdc4c57b4" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 43897 } } peers { permanent_uuid: "fd2272aee3234376a621b601a5739b44" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42299 } }
I20260929 16:46:40.569770 20086 leader_election.cc:288] T bb24460cfd5844dcaf821309598ef484 P 8f95a162b7f34cffbc5a311371ad49ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e788361fbd54b6590fdb0efdc4c57b4 (127.0.197.67:43897), fd2272aee3234376a621b601a5739b44 (127.0.197.66:42299)
I20260929 16:46:40.578675 20075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:40.570220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:37407 (local address 127.0.197.67:43897)
0929 16:46:40.570475 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:40.570486 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:40.570983 (+ 497us) server_negotiation.cc:407] Connection header received
0929 16:46:40.571247 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:40.571256 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:40.571360 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:40.571480 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:46:40.572157 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:40.573192 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:40.576040 (+ 2848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:40.577333 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:40.577499 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:40.577581 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:40.578255 (+ 674us) server_negotiation.cc:299] Negotiation successful
0929 16:46:40.578404 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 16:46:40.578691 20077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:40.570353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:42245 (local address 127.0.197.66:42299)
0929 16:46:40.570605 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:40.570613 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:40.571782 (+ 1169us) server_negotiation.cc:407] Connection header received
0929 16:46:40.571935 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:40.571939 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:40.572001 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:40.572067 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:46:40.572586 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:40.573504 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:40.576039 (+ 2535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:40.577279 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:40.577687 (+ 408us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:40.577759 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:40.578255 (+ 496us) server_negotiation.cc:299] Negotiation successful
0929 16:46:40.578404 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 16:46:40.578824 20088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:40.570866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.66:42299 (local address 127.0.197.65:42245)
0929 16:46:40.571655 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:40.571681 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 16:46:40.571805 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:46:40.572125 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:46:40.572130 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:46:40.572158 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:46:40.572410 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:40.572417 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:40.573635 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:40.573640 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:40.575739 (+ 2099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:40.575752 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:40.577658 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:40.577671 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:40.577946 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:40.578046 (+ 100us) client_negotiation.cc:769] Sending connection context
0929 16:46:40.578154 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 16:46:40.578405 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"mutex_wait_us":126,"thread_start_us":453,"threads_started":1}
I20260929 16:46:40.579133 20087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:40.570162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.67:43897 (local address 127.0.197.65:37407)
0929 16:46:40.570853 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:40.570869 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:46:40.571073 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:46:40.571533 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:46:40.571538 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:46:40.571574 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:46:40.571989 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:40.571999 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:40.573329 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:40.573332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:40.575734 (+ 2402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:40.575752 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:40.577690 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:40.577696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:40.577943 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:40.578051 (+ 108us) client_negotiation.cc:769] Sending connection context
0929 16:46:40.578154 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 16:46:40.578405 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"mutex_wait_us":165,"thread_start_us":345,"threads_started":1}
I20260929 16:46:40.580099 20015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb24460cfd5844dcaf821309598ef484" candidate_uuid: "8f95a162b7f34cffbc5a311371ad49ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e788361fbd54b6590fdb0efdc4c57b4" is_pre_election: true
I20260929 16:46:40.580099 19885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb24460cfd5844dcaf821309598ef484" candidate_uuid: "8f95a162b7f34cffbc5a311371ad49ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd2272aee3234376a621b601a5739b44" is_pre_election: true
I20260929 16:46:40.580729 19885 raft_consensus.cc:2454] T bb24460cfd5844dcaf821309598ef484 P fd2272aee3234376a621b601a5739b44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f95a162b7f34cffbc5a311371ad49ca in term 0.
I20260929 16:46:40.580727 20015 raft_consensus.cc:2454] T bb24460cfd5844dcaf821309598ef484 P 2e788361fbd54b6590fdb0efdc4c57b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f95a162b7f34cffbc5a311371ad49ca in term 0.
I20260929 16:46:40.581477 19687 leader_election.cc:302] T bb24460cfd5844dcaf821309598ef484 P 8f95a162b7f34cffbc5a311371ad49ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e788361fbd54b6590fdb0efdc4c57b4, 8f95a162b7f34cffbc5a311371ad49ca; no voters:
I20260929 16:46:40.582031 20086 raft_consensus.cc:2781] T bb24460cfd5844dcaf821309598ef484 P 8f95a162b7f34cffbc5a311371ad49ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:46:40.582185 20086 raft_consensus.cc:482] T bb24460cfd5844dcaf821309598ef484 P 8f95a162b7f34cffbc5a311371ad49ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:46:40.582286 20086 raft_consensus.cc:3037] T bb24460cfd5844dcaf821309598ef484 P 8f95a162b7f34cffbc5a311371ad49ca [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:46:40.584619 20086 raft_consensus.cc:504] T bb24460cfd5844dcaf821309598ef484 P 8f95a162b7f34cffbc5a311371ad49ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f95a162b7f34cffbc5a311371ad49ca" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 40017 } } peers { permanent_uuid: "2e788361fbd54b6590fdb0efdc4c57b4" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 43897 } } peers { permanent_uuid: "fd2272aee3234376a621b601a5739b44" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42299 } }
I20260929 16:46:40.585294 20086 leader_election.cc:288] T bb24460cfd5844dcaf821309598ef484 P 8f95a162b7f34cffbc5a311371ad49ca [CANDIDATE]: Term 1 election: Requested vote from peers 2e788361fbd54b6590fdb0efdc4c57b4 (127.0.197.67:43897), fd2272aee3234376a621b601a5739b44 (127.0.197.66:42299)
I20260929 16:46:40.585915 20015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb24460cfd5844dcaf821309598ef484" candidate_uuid: "8f95a162b7f34cffbc5a311371ad49ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e788361fbd54b6590fdb0efdc4c57b4"
I20260929 16:46:40.585915 19885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb24460cfd5844dcaf821309598ef484" candidate_uuid: "8f95a162b7f34cffbc5a311371ad49ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd2272aee3234376a621b601a5739b44"
I20260929 16:46:40.586161 20015 raft_consensus.cc:3037] T bb24460cfd5844dcaf821309598ef484 P 2e788361fbd54b6590fdb0efdc4c57b4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:46:40.586162 19885 raft_consensus.cc:3037] T bb24460cfd5844dcaf821309598ef484 P fd2272aee3234376a621b601a5739b44 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:46:40.589535 19885 raft_consensus.cc:2454] T bb24460cfd5844dcaf821309598ef484 P fd2272aee3234376a621b601a5739b44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f95a162b7f34cffbc5a311371ad49ca in term 1.
I20260929 16:46:40.589542 20015 raft_consensus.cc:2454] T bb24460cfd5844dcaf821309598ef484 P 2e788361fbd54b6590fdb0efdc4c57b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f95a162b7f34cffbc5a311371ad49ca in term 1.
I20260929 16:46:40.590132 19688 leader_election.cc:302] T bb24460cfd5844dcaf821309598ef484 P 8f95a162b7f34cffbc5a311371ad49ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2e788361fbd54b6590fdb0efdc4c57b4, 8f95a162b7f34cffbc5a311371ad49ca, fd2272aee3234376a621b601a5739b44; no voters:
I20260929 16:46:40.590512 20086 raft_consensus.cc:2781] T bb24460cfd5844dcaf821309598ef484 P 8f95a162b7f34cffbc5a311371ad49ca [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:46:40.591317 20086 raft_consensus.cc:686] T bb24460cfd5844dcaf821309598ef484 P 8f95a162b7f34cffbc5a311371ad49ca [term 1 LEADER]: Becoming Leader. State: Replica: 8f95a162b7f34cffbc5a311371ad49ca, State: Running, Role: LEADER
I20260929 16:46:40.591887 20086 consensus_queue.cc:237] T bb24460cfd5844dcaf821309598ef484 P 8f95a162b7f34cffbc5a311371ad49ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f95a162b7f34cffbc5a311371ad49ca" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 40017 } } peers { permanent_uuid: "2e788361fbd54b6590fdb0efdc4c57b4" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 43897 } } peers { permanent_uuid: "fd2272aee3234376a621b601a5739b44" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42299 } }
I20260929 16:46:40.596796 19612 catalog_manager.cc:5495] T bb24460cfd5844dcaf821309598ef484 P 8f95a162b7f34cffbc5a311371ad49ca reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f95a162b7f34cffbc5a311371ad49ca (127.0.197.65). New cstate: current_term: 1 leader_uuid: "8f95a162b7f34cffbc5a311371ad49ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f95a162b7f34cffbc5a311371ad49ca" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 40017 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e788361fbd54b6590fdb0efdc4c57b4" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 43897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd2272aee3234376a621b601a5739b44" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42299 } health_report { overall_health: UNKNOWN } } }
I20260929 16:46:40.666940 20074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:40.660131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38312 (local address 127.0.197.65:40017)
0929 16:46:40.660344 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:40.660349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:40.660370 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:46:40.660460 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:40.660463 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:40.660572 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:40.660718 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:46:40.661145 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:40.662279 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:40.664567 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:40.665120 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:40.665234 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:40.665781 (+ 547us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:46:40.666311 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:46:40.666442 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:40.666589 (+ 147us) server_negotiation.cc:299] Negotiation successful
0929 16:46:40.666630 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 16:46:40.675551 20015 raft_consensus.cc:1261] T bb24460cfd5844dcaf821309598ef484 P 2e788361fbd54b6590fdb0efdc4c57b4 [term 1 FOLLOWER]: Refusing update from remote peer 8f95a162b7f34cffbc5a311371ad49ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:46:40.675581 19885 raft_consensus.cc:1261] T bb24460cfd5844dcaf821309598ef484 P fd2272aee3234376a621b601a5739b44 [term 1 FOLLOWER]: Refusing update from remote peer 8f95a162b7f34cffbc5a311371ad49ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:46:40.676580 20086 consensus_queue.cc:1035] T bb24460cfd5844dcaf821309598ef484 P 8f95a162b7f34cffbc5a311371ad49ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e788361fbd54b6590fdb0efdc4c57b4" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 43897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:46:40.676903 20089 consensus_queue.cc:1035] T bb24460cfd5844dcaf821309598ef484 P 8f95a162b7f34cffbc5a311371ad49ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd2272aee3234376a621b601a5739b44" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 42299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:46:40.691715 20092 mvcc.cc:205] Tried to move back new op lower bound from 7334708841156280320 to 7334708840828477440. Current Snapshot: MvccSnapshot[applied={T|T < 7334708841156280320}]
I20260929 16:46:40.727350 19611 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:46:40.727691 19611 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:46:40.732149 19611 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 57d7597ca39d4c9ea2472fd2ba2e54db: Sending DeleteTablet for 3 replicas of tablet bb24460cfd5844dcaf821309598ef484
I20260929 16:46:40.733670 19995 tablet_service.cc:1514] Processing DeleteTablet for tablet bb24460cfd5844dcaf821309598ef484 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:46:40 UTC) from {username='slave'} at 127.0.0.1:55224
I20260929 16:46:40.733753 19734 tablet_service.cc:1514] Processing DeleteTablet for tablet bb24460cfd5844dcaf821309598ef484 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:46:40 UTC) from {username='slave'} at 127.0.0.1:38310
I20260929 16:46:40.733747 19865 tablet_service.cc:1514] Processing DeleteTablet for tablet bb24460cfd5844dcaf821309598ef484 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:46:40 UTC) from {username='slave'} at 127.0.0.1:42568
I20260929 16:46:40.736218 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 19672
I20260929 16:46:40.767108 20104 tablet_replica.cc:330] T bb24460cfd5844dcaf821309598ef484 P fd2272aee3234376a621b601a5739b44: stopping tablet replica
I20260929 16:46:40.767693 20104 raft_consensus.cc:2229] T bb24460cfd5844dcaf821309598ef484 P fd2272aee3234376a621b601a5739b44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:46:40.768265 20104 raft_consensus.cc:2258] T bb24460cfd5844dcaf821309598ef484 P fd2272aee3234376a621b601a5739b44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:46:40.769065 20102 tablet_replica.cc:330] T bb24460cfd5844dcaf821309598ef484 P 2e788361fbd54b6590fdb0efdc4c57b4: stopping tablet replica
I20260929 16:46:40.769544 20102 raft_consensus.cc:2229] T bb24460cfd5844dcaf821309598ef484 P 2e788361fbd54b6590fdb0efdc4c57b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:46:40.770095 20102 raft_consensus.cc:2258] T bb24460cfd5844dcaf821309598ef484 P 2e788361fbd54b6590fdb0efdc4c57b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:46:40.772151 20104 ts_tablet_manager.cc:1905] T bb24460cfd5844dcaf821309598ef484 P fd2272aee3234376a621b601a5739b44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:46:40.777273 20102 ts_tablet_manager.cc:1905] T bb24460cfd5844dcaf821309598ef484 P 2e788361fbd54b6590fdb0efdc4c57b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:46:40.779877 20074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:40.737833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38314 (local address 127.0.197.65:40017)
0929 16:46:40.738038 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:40.738045 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:40.761621 (+ 23576us) server_negotiation.cc:407] Connection header received
0929 16:46:40.762188 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:40.762196 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:40.762337 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:40.762435 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:40.763265 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:40.764533 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:40.770955 (+ 6422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:40.771755 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:40.771885 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:40.774262 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:40.774280 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:40.774300 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:46:40.775200 (+ 900us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:40.775281 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:46:40.775313 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:40.775435 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:40.779255 (+ 3820us) server_negotiation.cc:299] Negotiation successful
0929 16:46:40.779434 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 16:46:40.781340 19794 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:46:40.784423 20104 ts_tablet_manager.cc:1918] T bb24460cfd5844dcaf821309598ef484 P fd2272aee3234376a621b601a5739b44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:46:40.784832 20104 log.cc:1196] T bb24460cfd5844dcaf821309598ef484 P fd2272aee3234376a621b601a5739b44: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/ts-1/wal/wals/bb24460cfd5844dcaf821309598ef484
I20260929 16:46:40.785717 20104 ts_tablet_manager.cc:1939] T bb24460cfd5844dcaf821309598ef484 P fd2272aee3234376a621b601a5739b44: Deleting consensus metadata
I20260929 16:46:40.790530 20102 ts_tablet_manager.cc:1918] T bb24460cfd5844dcaf821309598ef484 P 2e788361fbd54b6590fdb0efdc4c57b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:46:40.790791 20102 log.cc:1196] T bb24460cfd5844dcaf821309598ef484 P 2e788361fbd54b6590fdb0efdc4c57b4: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790700068255611-789-0/minicluster-data/ts-2/wal/wals/bb24460cfd5844dcaf821309598ef484
I20260929 16:46:40.791735 20102 ts_tablet_manager.cc:1939] T bb24460cfd5844dcaf821309598ef484 P 2e788361fbd54b6590fdb0efdc4c57b4: Deleting consensus metadata
I20260929 16:46:40.794530 19597 catalog_manager.cc:4841] TS 2e788361fbd54b6590fdb0efdc4c57b4 (127.0.197.67:43897): tablet bb24460cfd5844dcaf821309598ef484 (table test-table [id=4b3f22eb6a5f4259b8121be228235ca4]) successfully deleted
I20260929 16:46:40.795986 19598 catalog_manager.cc:4841] TS fd2272aee3234376a621b601a5739b44 (127.0.197.66:42299): tablet bb24460cfd5844dcaf821309598ef484 (table test-table [id=4b3f22eb6a5f4259b8121be228235ca4]) successfully deleted
I20260929 16:46:40.898527 20103 tablet_replica.cc:330] T bb24460cfd5844dcaf821309598ef484 P 8f95a162b7f34cffbc5a311371ad49ca: stopping tablet replica
I20260929 16:46:40.899112 20103 raft_consensus.cc:2229] T bb24460cfd5844dcaf821309598ef484 P 8f95a162b7f34cffbc5a311371ad49ca [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:46:40.899861 20103 raft_consensus.cc:2258] T bb24460cfd5844dcaf821309598ef484 P 8f95a162b7f34cffbc5a311371ad49ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:46:40.900221 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 19672
I20260929 16:46:40.920029 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 19803
W20260929 16:46:40.920022 19600 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.197.65:40017 (error 108)
I20260929 16:46:40.921957 20108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:40.921127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:40017 (local address 127.0.0.1:38324)
0929 16:46:40.921525 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:40.921700 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.65:40017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":218,"threads_started":1}
W20260929 16:46:40.922220 19600 catalog_manager.cc:4584] TS 8f95a162b7f34cffbc5a311371ad49ca (127.0.197.65:40017): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bb24460cfd5844dcaf821309598ef484: Network error: Client connection negotiation failed: client connection to 127.0.197.65:40017: connect: Connection refused (error 111)
I20260929 16:46:40.931916 20077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:40.920838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42570 (local address 127.0.197.66:42299)
0929 16:46:40.921059 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:40.921066 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:40.921184 (+ 118us) server_negotiation.cc:407] Connection header received
0929 16:46:40.921445 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:40.921451 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:40.921576 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:40.921697 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:40.922205 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:40.923203 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:40.924552 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:40.924985 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:40.925089 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:40.930056 (+ 4967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:40.930068 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:40.930086 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:46:40.930747 (+ 661us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:40.930810 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:46:40.930836 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:40.931012 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:40.931309 (+ 297us) server_negotiation.cc:299] Negotiation successful
0929 16:46:40.931447 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260929 16:46:40.932751 19925 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:46:40.976190 20108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:40.975674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:40017 (local address 127.0.0.1:38326)
0929 16:46:40.975936 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:40.976034 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.65:40017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260929 16:46:40.977355 20108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:40.976957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:40017 (local address 127.0.0.1:38328)
0929 16:46:40.977146 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:40.977218 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.65:40017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 16:46:41.032555 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 19803
I20260929 16:46:41.037868 20108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:41.037390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:40017 (local address 127.0.0.1:38336)
0929 16:46:41.037611 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:41.037709 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.65:40017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260929 16:46:41.038938 20108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:41.038551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:40017 (local address 127.0.0.1:38338)
0929 16:46:41.038712 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:41.038793 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.65:40017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 16:46:41.050546 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 19933
I20260929 16:46:41.057595 20075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:41.051439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55230 (local address 127.0.197.67:43897)
0929 16:46:41.051661 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:41.051666 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:41.051686 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:46:41.051770 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:41.051774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:41.051862 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:41.051987 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:41.052372 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:41.053169 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:41.054434 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:41.054864 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:41.054971 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:41.056153 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:41.056161 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:41.056175 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:46:41.056791 (+ 616us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:41.056841 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:46:41.056863 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:41.056956 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:41.057092 (+ 136us) server_negotiation.cc:299] Negotiation successful
0929 16:46:41.057192 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260929 16:46:41.058311 20055 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:46:41.119921 20108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:41.119335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:40017 (local address 127.0.0.1:38352)
0929 16:46:41.119586 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:41.119730 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.65:40017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260929 16:46:41.121214 20108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:41.120728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:40017 (local address 127.0.0.1:38354)
0929 16:46:41.120954 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:41.121044 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.65:40017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260929 16:46:41.150563 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 19933
I20260929 16:46:41.168255 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 19582
I20260929 16:46:41.169492 19643 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:46:41.282810 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 19582
2026-09-29T16:46:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3136 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:46:41 dist-test-slave-l31f krb5kdc[20114](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:46:41 dist-test-slave-l31f krb5kdc[20114](info): set up 2 sockets
Sep 29 16:46:41 dist-test-slave-l31f krb5kdc[20114](info): commencing operation
krb5kdc: starting...
W20260929 16:46:42.419433 789 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.002s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/krb5kdc/test-user.keytab.
Sep 29 16:46:42 dist-test-slave-l31f krb5kdc[20114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:46:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:46:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.197.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
I20260929 16:46:42.699682 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:44915
--webserver_interface=127.0.197.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:41913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.126:44915
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:46:43.059681 20127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:46:43.060101 20127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:46:43.060273 20127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:46:43.067970 20127 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:46:43.068203 20127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:46:43.068328 20127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:46:43.068434 20127 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:46:43.068668 20127 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:46:43.079324 20127 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:41913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.197.126:44915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:44915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.197.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:46:43.080850 20127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:46:43.082415 20127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:46:43.095678 20133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:43.095816 20135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:43.095970 20132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:46:43.096681 20127 server_base.cc:1034] running on GCE node
I20260929 16:46:43.097563 20127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:46:43.099298 20127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:46:43.100605 20127 hybrid_clock.cc:648] HybridClock initialized: now 1790700403100556 us; error 71 us; skew 500 ppm
I20260929 16:46:43.101069 20127 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:46:43 dist-test-slave-l31f krb5kdc[20114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:46:43.105002 20127 init.cc:377] Logged in from keytab as kudu/127.0.197.126@KRBTEST.COM (short username kudu)
I20260929 16:46:43.107100 20127 webserver.cc:460] Webserver started at http://127.0.197.126:45649/ using document root <none> and password file <none>
I20260929 16:46:43.107882 20127 fs_manager.cc:359] Metadata directory not provided
I20260929 16:46:43.108047 20127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:46:43.108393 20127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:46:43.111761 20127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/master-0/data/instance:
uuid: "2a961cd3f91149b7af4fe109223c2365"
format_stamp: "Formatted at 2026-09-29 16:46:43 on dist-test-slave-l31f"
server_key: "f655ebfc1a44d247f5e9267d79c37204"
server_key_iv: "1b4f3053ac4557ced8bdcdd5a630540e"
server_key_version: "encryptionkey@0"
I20260929 16:46:43.112685 20127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/master-0/wal/instance:
uuid: "2a961cd3f91149b7af4fe109223c2365"
format_stamp: "Formatted at 2026-09-29 16:46:43 on dist-test-slave-l31f"
server_key: "f655ebfc1a44d247f5e9267d79c37204"
server_key_iv: "1b4f3053ac4557ced8bdcdd5a630540e"
server_key_version: "encryptionkey@0"
I20260929 16:46:43.119107 20127 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 16:46:43.123023 20142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:43.124702 20127 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 16:46:43.124954 20127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "2a961cd3f91149b7af4fe109223c2365"
format_stamp: "Formatted at 2026-09-29 16:46:43 on dist-test-slave-l31f"
server_key: "f655ebfc1a44d247f5e9267d79c37204"
server_key_iv: "1b4f3053ac4557ced8bdcdd5a630540e"
server_key_version: "encryptionkey@0"
I20260929 16:46:43.125193 20127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:46:43.151187 20127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:46:43.156136 20127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:46:43.156454 20127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:46:43.178342 20127 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.126:44915
I20260929 16:46:43.178339 20193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.126:44915 every 8 connection(s)
I20260929 16:46:43.179980 20127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:46:43.184366 20194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:46:43.187973 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 20127
I20260929 16:46:43.188167 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/master-0/wal/instance
I20260929 16:46:43.188694 789 external_mini_cluster.cc:1428] Setting key dc7fc1d6306ef86ddfc30c5753e9582e
I20260929 16:46:43.198248 20194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a961cd3f91149b7af4fe109223c2365: Bootstrap starting.
I20260929 16:46:43.202663 20194 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a961cd3f91149b7af4fe109223c2365: Neither blocks nor log segments found. Creating new log.
I20260929 16:46:43.204277 20194 log.cc:824] T 00000000000000000000000000000000 P 2a961cd3f91149b7af4fe109223c2365: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:46:43 dist-test-slave-l31f krb5kdc[20114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:46:43.208338 20194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a961cd3f91149b7af4fe109223c2365: No bootstrap required, opened a new log
I20260929 16:46:43.216310 20197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:43.190777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59020 (local address 127.0.197.126:44915)
0929 16:46:43.191692 (+ 915us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:43.191706 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:43.191752 (+ 46us) server_negotiation.cc:407] Connection header received
0929 16:46:43.193495 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:43.193523 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:43.193977 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:43.194329 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:43.196531 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:43.198162 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:43.200335 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:43.201125 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:43.201284 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:43.206990 (+ 5706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:43.207010 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:43.207127 (+ 117us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:46:43.207184 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:43.211205 (+ 4021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:43.212264 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:43.212272 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:43.212280 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:43.212505 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:43.212982 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:43.212986 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:43.212988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:43.213672 (+ 684us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:43.213923 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:43.214515 (+ 592us) server_negotiation.cc:299] Negotiation successful
0929 16:46:43.215107 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":694,"thread_start_us":439,"threads_started":1}
I20260929 16:46:43.217716 20194 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a961cd3f91149b7af4fe109223c2365 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a961cd3f91149b7af4fe109223c2365" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 44915 } }
I20260929 16:46:43.218119 20194 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a961cd3f91149b7af4fe109223c2365 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:46:43.218259 20194 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a961cd3f91149b7af4fe109223c2365 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a961cd3f91149b7af4fe109223c2365, State: Initialized, Role: FOLLOWER
I20260929 16:46:43.218940 20194 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a961cd3f91149b7af4fe109223c2365 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a961cd3f91149b7af4fe109223c2365" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 44915 } }
I20260929 16:46:43.219276 20194 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a961cd3f91149b7af4fe109223c2365 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:46:43.219425 20194 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a961cd3f91149b7af4fe109223c2365 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:46:43.219573 20194 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a961cd3f91149b7af4fe109223c2365 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:46:43.221871 20194 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a961cd3f91149b7af4fe109223c2365 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a961cd3f91149b7af4fe109223c2365" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 44915 } }
I20260929 16:46:43.222599 20194 leader_election.cc:302] T 00000000000000000000000000000000 P 2a961cd3f91149b7af4fe109223c2365 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a961cd3f91149b7af4fe109223c2365; no voters:
I20260929 16:46:43.223367 20194 leader_election.cc:288] T 00000000000000000000000000000000 P 2a961cd3f91149b7af4fe109223c2365 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:46:43.223510 20199 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a961cd3f91149b7af4fe109223c2365 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:46:43.224349 20199 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a961cd3f91149b7af4fe109223c2365 [term 1 LEADER]: Becoming Leader. State: Replica: 2a961cd3f91149b7af4fe109223c2365, State: Running, Role: LEADER
I20260929 16:46:43.224926 20199 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a961cd3f91149b7af4fe109223c2365 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a961cd3f91149b7af4fe109223c2365" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 44915 } }
I20260929 16:46:43.225837 20194 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a961cd3f91149b7af4fe109223c2365 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:46:43.228333 20201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a961cd3f91149b7af4fe109223c2365 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a961cd3f91149b7af4fe109223c2365. Latest consensus state: current_term: 1 leader_uuid: "2a961cd3f91149b7af4fe109223c2365" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a961cd3f91149b7af4fe109223c2365" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 44915 } } }
I20260929 16:46:43.228360 20200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a961cd3f91149b7af4fe109223c2365 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a961cd3f91149b7af4fe109223c2365" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a961cd3f91149b7af4fe109223c2365" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 44915 } } }
I20260929 16:46:43.228801 20201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a961cd3f91149b7af4fe109223c2365 [sys.catalog]: This master's current role is: LEADER
I20260929 16:46:43.228834 20200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a961cd3f91149b7af4fe109223c2365 [sys.catalog]: This master's current role is: LEADER
I20260929 16:46:43.235091 20204 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:46:43.241147 20204 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:46:43.251824 20204 catalog_manager.cc:1347] Generated new cluster ID: 56882eac06684c9e93438ba45b1cfbbd
I20260929 16:46:43.251987 20204 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:46:43.264418 20204 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:46:43.266019 20204 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:46:43.272810 20204 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a961cd3f91149b7af4fe109223c2365: Generated new TSK 0
I20260929 16:46:43.273821 20204 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.197.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab.
Entry for principal kudu/127.0.197.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab.
I20260929 16:46:43.351346 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.65:0
--local_ip_for_outbound_sockets=127.0.197.65
--webserver_interface=127.0.197.65
--webserver_port=0
--tserver_master_addrs=127.0.197.126:44915
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab
--principal=kudu/127.0.197.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:41913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:46:43.721369 20220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:46:43.721680 20220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:46:43.721779 20220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:46:43.729831 20220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:46:43.729965 20220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:46:43.730100 20220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.65
I20260929 16:46:43.740938 20220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:41913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/krb5kdc/kudu_127.0.197.65.keytab
--principal=kudu/127.0.197.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.197.65
--webserver_port=0
--tserver_master_addrs=127.0.197.126:44915
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.65
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:46:43.742376 20220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:46:43.743940 20220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:46:43.758661 20228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:43.758874 20226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:43.758796 20225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:46:43.759752 20220 server_base.cc:1034] running on GCE node
I20260929 16:46:43.760510 20220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:46:43.761682 20220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:46:43.763109 20220 hybrid_clock.cc:648] HybridClock initialized: now 1790700403762958 us; error 84 us; skew 500 ppm
I20260929 16:46:43.763505 20220 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:46:43 dist-test-slave-l31f krb5kdc[20114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:46:43.767354 20220 init.cc:377] Logged in from keytab as kudu/127.0.197.65@KRBTEST.COM (short username kudu)
I20260929 16:46:43.769116 20220 webserver.cc:460] Webserver started at http://127.0.197.65:32783/ using document root <none> and password file <none>
I20260929 16:46:43.769870 20220 fs_manager.cc:359] Metadata directory not provided
I20260929 16:46:43.769985 20220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:46:43.770275 20220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:46:43.773586 20220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/ts-0/data/instance:
uuid: "f518e988785a4161b3e9c7a39effdf45"
format_stamp: "Formatted at 2026-09-29 16:46:43 on dist-test-slave-l31f"
server_key: "fcf1fdbed87c386d1d80c7356d378490"
server_key_iv: "76548a6f8d4edc6180c2575fc63b764d"
server_key_version: "encryptionkey@0"
I20260929 16:46:43.774434 20220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/ts-0/wal/instance:
uuid: "f518e988785a4161b3e9c7a39effdf45"
format_stamp: "Formatted at 2026-09-29 16:46:43 on dist-test-slave-l31f"
server_key: "fcf1fdbed87c386d1d80c7356d378490"
server_key_iv: "76548a6f8d4edc6180c2575fc63b764d"
server_key_version: "encryptionkey@0"
I20260929 16:46:43.780378 20220 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260929 16:46:43.784610 20235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:43.786518 20220 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:46:43.786713 20220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/ts-0/wal
uuid: "f518e988785a4161b3e9c7a39effdf45"
format_stamp: "Formatted at 2026-09-29 16:46:43 on dist-test-slave-l31f"
server_key: "fcf1fdbed87c386d1d80c7356d378490"
server_key_iv: "76548a6f8d4edc6180c2575fc63b764d"
server_key_version: "encryptionkey@0"
I20260929 16:46:43.786939 20220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:46:43.815164 20220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:46:43.820549 20220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:46:43.820843 20220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:46:43.821918 20220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:46:43.823530 20220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:46:43.823618 20220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:43.823717 20220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:46:43.823766 20220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:43.862591 20220 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.65:37639
I20260929 16:46:43.862705 20347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.65:37639 every 8 connection(s)
I20260929 16:46:43.864205 20220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
I20260929 16:46:43.874406 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 20220
I20260929 16:46:43.874569 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/ts-0/wal/instance
I20260929 16:46:43.875136 789 external_mini_cluster.cc:1428] Setting key d6dbd794f256124737aaed1f471daeba
Sep 29 16:46:43 dist-test-slave-l31f krb5kdc[20114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.65@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:46:43.886389 20350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:43.867087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:49623 (local address 127.0.197.126:44915)
0929 16:46:43.867570 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:43.867575 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:43.868478 (+ 903us) server_negotiation.cc:407] Connection header received
0929 16:46:43.870476 (+ 1998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:43.870481 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:43.870589 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:43.870722 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:43.872424 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:43.873161 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:43.874964 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:43.875531 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:43.875686 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:43.879804 (+ 4118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:43.879809 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:43.879812 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:46:43.879850 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:43.882722 (+ 2872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:43.883755 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:43.883760 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:43.883762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:43.883919 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:43.884421 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:43.884432 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:43.884435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:43.884743 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:43.884931 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:43.885741 (+ 810us) server_negotiation.cc:299] Negotiation successful
0929 16:46:43.886018 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":258,"threads_started":1}
I20260929 16:46:43.887614 20351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:43.867561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:44915 (local address 127.0.197.65:49623)
0929 16:46:43.868305 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:43.868346 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 16:46:43.870143 (+ 1797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:46:43.870980 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:46:43.870989 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:46:43.871615 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:46:43.872226 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:43.872246 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:43.873346 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:43.873350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:43.874801 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:43.874815 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:43.875797 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:43.875805 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:43.876068 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:43.876824 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:46:43.876848 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:46:43.879526 (+ 2678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:46:43.882960 (+ 3434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:46:43.882973 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:46:43.882995 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:46:43.883588 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:46:43.884056 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:46:43.884061 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:46:43.884065 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:46:43.884285 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:46:43.884897 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:46:43.884906 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:46:43.885305 (+ 399us) client_negotiation.cc:769] Sending connection context
0929 16:46:43.885603 (+ 298us) client_negotiation.cc:240] Negotiation successful
0929 16:46:43.886202 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":293,"threads_started":1}
I20260929 16:46:43.889321 20348 heartbeater.cc:347] Connected to a master server at 127.0.197.126:44915
I20260929 16:46:43.889678 20348 heartbeater.cc:464] Registering TS with master...
I20260929 16:46:43.890463 20348 heartbeater.cc:511] Master 127.0.197.126:44915 requested a full tablet report, sending...
I20260929 16:46:43.893070 20159 ts_manager.cc:194] Registered new tserver with Master: f518e988785a4161b3e9c7a39effdf45 (127.0.197.65:37639)
I20260929 16:46:43.894680 20159 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.65@KRBTEST.COM'} at 127.0.197.65:49623
WARNING: no policy specified for kudu/127.0.197.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab.
Entry for principal kudu/127.0.197.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab.
I20260929 16:46:43.935343 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.66:0
--local_ip_for_outbound_sockets=127.0.197.66
--webserver_interface=127.0.197.66
--webserver_port=0
--tserver_master_addrs=127.0.197.126:44915
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab
--principal=kudu/127.0.197.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:41913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:46:44.287190 20354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:46:44.287508 20354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:46:44.287606 20354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:46:44.295739 20354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:46:44.295877 20354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:46:44.296007 20354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.66
I20260929 16:46:44.306350 20354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:41913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/krb5kdc/kudu_127.0.197.66.keytab
--principal=kudu/127.0.197.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.197.66
--webserver_port=0
--tserver_master_addrs=127.0.197.126:44915
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.66
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:46:44.307879 20354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:46:44.309199 20354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:46:44.323490 20362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:44.323581 20359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:44.323540 20360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:46:44.324777 20354 server_base.cc:1034] running on GCE node
I20260929 16:46:44.325606 20354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:46:44.326812 20354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:46:44.328087 20354 hybrid_clock.cc:648] HybridClock initialized: now 1790700404327994 us; error 84 us; skew 500 ppm
I20260929 16:46:44.328495 20354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:46:44 dist-test-slave-l31f krb5kdc[20114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:46:44.331959 20354 init.cc:377] Logged in from keytab as kudu/127.0.197.66@KRBTEST.COM (short username kudu)
I20260929 16:46:44.333580 20354 webserver.cc:460] Webserver started at http://127.0.197.66:42839/ using document root <none> and password file <none>
I20260929 16:46:44.334316 20354 fs_manager.cc:359] Metadata directory not provided
I20260929 16:46:44.334429 20354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:46:44.334698 20354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:46:44.337821 20354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/ts-1/data/instance:
uuid: "e4cbfb5e73004cc5867df1c6e836d5ee"
format_stamp: "Formatted at 2026-09-29 16:46:44 on dist-test-slave-l31f"
server_key: "ab361d0c1f92c5ca105fedd342036b4a"
server_key_iv: "9fab1a2d7161165c8505eb8bf645be4d"
server_key_version: "encryptionkey@0"
I20260929 16:46:44.338637 20354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/ts-1/wal/instance:
uuid: "e4cbfb5e73004cc5867df1c6e836d5ee"
format_stamp: "Formatted at 2026-09-29 16:46:44 on dist-test-slave-l31f"
server_key: "ab361d0c1f92c5ca105fedd342036b4a"
server_key_iv: "9fab1a2d7161165c8505eb8bf645be4d"
server_key_version: "encryptionkey@0"
I20260929 16:46:44.344763 20354 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 16:46:44.348758 20369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:44.350389 20354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260929 16:46:44.350621 20354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/ts-1/wal
uuid: "e4cbfb5e73004cc5867df1c6e836d5ee"
format_stamp: "Formatted at 2026-09-29 16:46:44 on dist-test-slave-l31f"
server_key: "ab361d0c1f92c5ca105fedd342036b4a"
server_key_iv: "9fab1a2d7161165c8505eb8bf645be4d"
server_key_version: "encryptionkey@0"
I20260929 16:46:44.350835 20354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:46:44.367367 20354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:46:44.372807 20354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:46:44.373160 20354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:46:44.374266 20354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:46:44.375810 20354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:46:44.375891 20354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:44.376049 20354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:46:44.376101 20354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:44.420346 20354 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.66:36589
I20260929 16:46:44.420385 20481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.66:36589 every 8 connection(s)
I20260929 16:46:44.421756 20354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
I20260929 16:46:44.423359 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 20354
I20260929 16:46:44.423568 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/ts-1/wal/instance
I20260929 16:46:44.424062 789 external_mini_cluster.cc:1428] Setting key 811c372635b8efe03a75c7f968294160
Sep 29 16:46:44 dist-test-slave-l31f krb5kdc[20114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.66@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.197.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.67@KRBTEST.COM" created.
I20260929 16:46:44.462773 20485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:44.426420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:51485 (local address 127.0.197.126:44915)
0929 16:46:44.431473 (+ 5053us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:44.431483 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:44.435778 (+ 4295us) server_negotiation.cc:407] Connection header received
0929 16:46:44.439088 (+ 3310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:44.439097 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:44.439211 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:44.439341 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:44.441743 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:44.443180 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:44.446191 (+ 3011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:44.447085 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:44.447222 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:44.455666 (+ 8444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:44.455675 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:44.455680 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:46:44.455734 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:44.459219 (+ 3485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:44.460160 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:44.460167 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:44.460170 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:44.460317 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:44.460823 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:44.460829 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:44.460832 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:44.461136 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:44.461323 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:44.462104 (+ 781us) server_negotiation.cc:299] Negotiation successful
0929 16:46:44.462363 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4887,"thread_start_us":4758,"threads_started":1}
I20260929 16:46:44.464010 20484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:44.425809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:44915 (local address 127.0.197.66:51485)
0929 16:46:44.435478 (+ 9669us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:44.435571 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0929 16:46:44.438596 (+ 3025us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:46:44.439694 (+ 1098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:46:44.439707 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:46:44.440550 (+ 843us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:46:44.441461 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:44.441493 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:44.443441 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:44.443451 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:44.445939 (+ 2488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:44.445964 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:44.447661 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:44.447671 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:44.448086 (+ 415us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:44.451962 (+ 3876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:46:44.452000 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:46:44.455250 (+ 3250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:46:44.459447 (+ 4197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:46:44.459462 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:46:44.459479 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:46:44.459999 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:46:44.460464 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:46:44.460469 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:46:44.460473 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:46:44.460697 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:46:44.461295 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:46:44.461304 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:46:44.461711 (+ 407us) client_negotiation.cc:769] Sending connection context
0929 16:46:44.462020 (+ 309us) client_negotiation.cc:240] Negotiation successful
0929 16:46:44.462623 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":624,"threads_started":1}
I20260929 16:46:44.466218 20482 heartbeater.cc:347] Connected to a master server at 127.0.197.126:44915
I20260929 16:46:44.466598 20482 heartbeater.cc:464] Registering TS with master...
I20260929 16:46:44.467482 20482 heartbeater.cc:511] Master 127.0.197.126:44915 requested a full tablet report, sending...
I20260929 16:46:44.469136 20159 ts_manager.cc:194] Registered new tserver with Master: e4cbfb5e73004cc5867df1c6e836d5ee (127.0.197.66:36589)
I20260929 16:46:44.470021 20159 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.66@KRBTEST.COM'} at 127.0.197.66:51485
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab.
Entry for principal kudu/127.0.197.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab.
I20260929 16:46:44.495201 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.67:0
--local_ip_for_outbound_sockets=127.0.197.67
--webserver_interface=127.0.197.67
--webserver_port=0
--tserver_master_addrs=127.0.197.126:44915
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab
--principal=kudu/127.0.197.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:41913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:46:44.849606 20488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:46:44.849980 20488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:46:44.850174 20488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:46:44.858247 20488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:46:44.858441 20488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:46:44.858635 20488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.67
I20260929 16:46:44.869450 20488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:41913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/krb5kdc/kudu_127.0.197.67.keytab
--principal=kudu/127.0.197.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.197.67
--webserver_port=0
--tserver_master_addrs=127.0.197.126:44915
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.67
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:46:44.870896 20488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:46:44.872347 20488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:46:44.887091 20494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:44.887117 20496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:44.887176 20493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:46:44.887948 20488 server_base.cc:1034] running on GCE node
I20260929 16:46:44.888685 20488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:46:44.889777 20488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:46:44.891100 20488 hybrid_clock.cc:648] HybridClock initialized: now 1790700404891022 us; error 66 us; skew 500 ppm
I20260929 16:46:44.891501 20488 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:46:44 dist-test-slave-l31f krb5kdc[20114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:46:44.895203 20488 init.cc:377] Logged in from keytab as kudu/127.0.197.67@KRBTEST.COM (short username kudu)
I20260929 16:46:44.896976 20488 webserver.cc:460] Webserver started at http://127.0.197.67:37357/ using document root <none> and password file <none>
I20260929 16:46:44.897806 20488 fs_manager.cc:359] Metadata directory not provided
I20260929 16:46:44.897981 20488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:46:44.898038 20348 heartbeater.cc:503] Master 127.0.197.126:44915 was elected leader, sending a full tablet report...
I20260929 16:46:44.898399 20488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:46:44.902046 20488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/ts-2/data/instance:
uuid: "c7ac916426934c799550e67f33615133"
format_stamp: "Formatted at 2026-09-29 16:46:44 on dist-test-slave-l31f"
server_key: "c1a01d605304fe34f2e71c37ec5b8464"
server_key_iv: "468da6f52d5ee234793ec3444fe1acb2"
server_key_version: "encryptionkey@0"
I20260929 16:46:44.902987 20488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/ts-2/wal/instance:
uuid: "c7ac916426934c799550e67f33615133"
format_stamp: "Formatted at 2026-09-29 16:46:44 on dist-test-slave-l31f"
server_key: "c1a01d605304fe34f2e71c37ec5b8464"
server_key_iv: "468da6f52d5ee234793ec3444fe1acb2"
server_key_version: "encryptionkey@0"
I20260929 16:46:44.908836 20488 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 16:46:44.912941 20503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:44.914625 20488 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:46:44.914844 20488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/ts-2/wal
uuid: "c7ac916426934c799550e67f33615133"
format_stamp: "Formatted at 2026-09-29 16:46:44 on dist-test-slave-l31f"
server_key: "c1a01d605304fe34f2e71c37ec5b8464"
server_key_iv: "468da6f52d5ee234793ec3444fe1acb2"
server_key_version: "encryptionkey@0"
I20260929 16:46:44.915143 20488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:46:44.929497 20488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:46:44.934304 20488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:46:44.934644 20488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:46:44.935796 20488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:46:44.937415 20488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:46:44.937522 20488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:44.937660 20488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:46:44.937713 20488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:44.973716 20488 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.67:42959
I20260929 16:46:44.973780 20615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.67:42959 every 8 connection(s)
I20260929 16:46:44.975198 20488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
I20260929 16:46:44.982836 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 20488
I20260929 16:46:44.983095 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/ts-2/wal/instance
I20260929 16:46:44.983572 789 external_mini_cluster.cc:1428] Setting key eb8a374a792ed41ed8cd361dc671ae4e
Sep 29 16:46:44 dist-test-slave-l31f krb5kdc[20114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.67@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:46:45.001216 20618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:44.978088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:53379 (local address 127.0.197.126:44915)
0929 16:46:44.978576 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:44.978581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:44.979641 (+ 1060us) server_negotiation.cc:407] Connection header received
0929 16:46:44.981659 (+ 2018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:44.981665 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:44.981771 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:44.981903 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:44.983851 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:44.984851 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:44.987155 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:44.987581 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:44.987705 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:44.991892 (+ 4187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:44.991898 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:44.991901 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:46:44.991941 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:44.994578 (+ 2637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:44.995888 (+ 1310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:44.995897 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:44.995902 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:44.996091 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:44.997022 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:44.997028 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:44.997032 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:44.997648 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:44.998192 (+ 544us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:45.000169 (+ 1977us) server_negotiation.cc:299] Negotiation successful
0929 16:46:45.000467 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":269,"threads_started":1}
I20260929 16:46:45.003315 20619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:44.978633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:44915 (local address 127.0.197.67:53379)
0929 16:46:44.979444 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:44.979493 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 16:46:44.981306 (+ 1813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:46:44.982174 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:46:44.982183 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:46:44.982786 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:46:44.983602 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:44.983653 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:44.985082 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:44.985089 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:44.986917 (+ 1828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:44.986941 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:44.987814 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:44.987819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:44.988045 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:44.988785 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:46:44.988809 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:46:44.991596 (+ 2787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:46:44.994814 (+ 3218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:46:44.994825 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:46:44.994852 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:46:44.995695 (+ 843us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:46:44.996252 (+ 557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:46:44.996260 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:46:44.996267 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:46:44.996819 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:46:44.998281 (+ 1462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:46:44.998292 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:46:44.999157 (+ 865us) client_negotiation.cc:769] Sending connection context
0929 16:46:45.000801 (+ 1644us) client_negotiation.cc:240] Negotiation successful
0929 16:46:45.001565 (+ 764us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":365,"threads_started":1}
I20260929 16:46:45.005409 20616 heartbeater.cc:347] Connected to a master server at 127.0.197.126:44915
I20260929 16:46:45.005849 20616 heartbeater.cc:464] Registering TS with master...
I20260929 16:46:45.006942 20616 heartbeater.cc:511] Master 127.0.197.126:44915 requested a full tablet report, sending...
I20260929 16:46:45.009548 20159 ts_manager.cc:194] Registered new tserver with Master: c7ac916426934c799550e67f33615133 (127.0.197.67:42959)
I20260929 16:46:45.011204 20159 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.197.67@KRBTEST.COM'} at 127.0.197.67:53379
I20260929 16:46:45.013520 789 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:46:45.034121 20618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:45.018508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55052 (local address 127.0.197.126:44915)
0929 16:46:45.018712 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:45.018718 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:45.019683 (+ 965us) server_negotiation.cc:407] Connection header received
0929 16:46:45.019891 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:45.019896 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:45.019977 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:45.020139 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:45.021705 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:45.022543 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:45.024504 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:45.024959 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:45.025094 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:45.027964 (+ 2870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:45.027970 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:45.027974 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:46:45.028016 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:45.031211 (+ 3195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:45.031966 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:45.031971 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:45.031973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:45.032105 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:45.032595 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:45.032599 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:45.032601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:45.032966 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:45.033164 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:45.033537 (+ 373us) server_negotiation.cc:299] Negotiation successful
0929 16:46:45.033732 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 16:46:45.038789 20158 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55052:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:46:45.046221 20158 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:46:45.079311 20630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:45.060673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47008 (local address 127.0.197.65:37639)
0929 16:46:45.062983 (+ 2310us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:45.062997 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:45.063073 (+ 76us) server_negotiation.cc:407] Connection header received
0929 16:46:45.063208 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:45.063221 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:45.065176 (+ 1955us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:45.066554 (+ 1378us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:46:45.066864 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:45.070773 (+ 3909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:45.076109 (+ 5336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:45.077715 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:45.077939 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:45.078244 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:45.078718 (+ 474us) server_negotiation.cc:299] Negotiation successful
0929 16:46:45.078907 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2068,"thread_start_us":1849,"threads_started":1}
I20260929 16:46:45.079360 20631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:45.060385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:37639 (local address 127.0.0.1:47008)
0929 16:46:45.061948 (+ 1563us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:45.061990 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 16:46:45.062203 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:46:45.065527 (+ 3324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:46:45.065538 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:46:45.065593 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:46:45.066223 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:45.066242 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:45.071069 (+ 4827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:45.071080 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:45.074340 (+ 3260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:45.074364 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:45.078022 (+ 3658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:45.078034 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:45.078378 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:45.078470 (+ 92us) client_negotiation.cc:769] Sending connection context
0929 16:46:45.078641 (+ 171us) client_negotiation.cc:240] Negotiation successful
0929 16:46:45.078872 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1290,"mutex_wait_us":5,"thread_start_us":2616,"threads_started":1}
I20260929 16:46:45.083985 20283 tablet_service.cc:1467] Processing CreateTablet for tablet 40e0f419294943aba52f41c6faa1c45c (DEFAULT_TABLE table=test-table [id=571330147bd840a8af10589999300703]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:46:45.085662 20633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:45.063662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.67:42959 (local address 127.0.0.1:50996)
0929 16:46:45.076806 (+ 13144us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:45.076822 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:46:45.076986 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:46:45.077601 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:46:45.077606 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:46:45.077636 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:46:45.078100 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:45.078111 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:45.081178 (+ 3067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:45.081183 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:45.083119 (+ 1936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:45.083141 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:45.084910 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:45.084919 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:45.085086 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:45.085114 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 16:46:45.085201 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 16:46:45.085350 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13017,"thread_start_us":12874,"threads_started":1}
I20260929 16:46:45.085875 20283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40e0f419294943aba52f41c6faa1c45c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:46:45.086149 20634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:45.063754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50996 (local address 127.0.197.67:42959)
0929 16:46:45.076133 (+ 12379us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:45.076145 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:45.076942 (+ 797us) server_negotiation.cc:407] Connection header received
0929 16:46:45.077169 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:45.077176 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:45.077394 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:45.077612 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:46:45.078357 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:45.080965 (+ 2608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:45.083350 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:45.084324 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:45.084521 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:45.084756 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:45.085293 (+ 537us) server_negotiation.cc:299] Negotiation successful
0929 16:46:45.085496 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12117,"thread_start_us":12409,"threads_started":1}
I20260929 16:46:45.086227 20629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:45.060973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.66:36589 (local address 127.0.0.1:50154)
0929 16:46:45.063277 (+ 2304us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:45.063298 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:46:45.063420 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:46:45.075418 (+ 11998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:46:45.075423 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:46:45.075450 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:46:45.075927 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:45.075944 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:45.078062 (+ 2118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:45.078068 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:45.080275 (+ 2207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:45.080295 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:45.084575 (+ 4280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:45.084582 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:45.084782 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:45.084821 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 16:46:45.084951 (+ 130us) client_negotiation.cc:240] Negotiation successful
0929 16:46:45.085112 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2173,"mutex_wait_us":25,"thread_start_us":715,"threads_started":1}
I20260929 16:46:45.087440 20632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:45.061482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50154 (local address 127.0.197.66:36589)
0929 16:46:45.074420 (+ 12938us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:45.074433 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:45.074468 (+ 35us) server_negotiation.cc:407] Connection header received
0929 16:46:45.074597 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:45.074605 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:45.074922 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:45.076150 (+ 1228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:46:45.076375 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:45.077896 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:45.080484 (+ 2588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:45.084283 (+ 3799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:45.084479 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:45.084714 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:45.086798 (+ 2084us) server_negotiation.cc:299] Negotiation successful
0929 16:46:45.087029 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12723,"thread_start_us":13631,"threads_started":1}
I20260929 16:46:45.098423 20551 tablet_service.cc:1467] Processing CreateTablet for tablet 40e0f419294943aba52f41c6faa1c45c (DEFAULT_TABLE table=test-table [id=571330147bd840a8af10589999300703]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:46:45.098415 20417 tablet_service.cc:1467] Processing CreateTablet for tablet 40e0f419294943aba52f41c6faa1c45c (DEFAULT_TABLE table=test-table [id=571330147bd840a8af10589999300703]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:46:45.100169 20551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40e0f419294943aba52f41c6faa1c45c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:46:45.100172 20417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40e0f419294943aba52f41c6faa1c45c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:46:45.117076 20635 tablet_bootstrap.cc:492] T 40e0f419294943aba52f41c6faa1c45c P f518e988785a4161b3e9c7a39effdf45: Bootstrap starting.
I20260929 16:46:45.119396 20637 tablet_bootstrap.cc:492] T 40e0f419294943aba52f41c6faa1c45c P e4cbfb5e73004cc5867df1c6e836d5ee: Bootstrap starting.
I20260929 16:46:45.122766 20635 tablet_bootstrap.cc:654] T 40e0f419294943aba52f41c6faa1c45c P f518e988785a4161b3e9c7a39effdf45: Neither blocks nor log segments found. Creating new log.
I20260929 16:46:45.124167 20637 tablet_bootstrap.cc:654] T 40e0f419294943aba52f41c6faa1c45c P e4cbfb5e73004cc5867df1c6e836d5ee: Neither blocks nor log segments found. Creating new log.
I20260929 16:46:45.124854 20635 log.cc:824] T 40e0f419294943aba52f41c6faa1c45c P f518e988785a4161b3e9c7a39effdf45: Log is configured to *not* fsync() on all Append() calls
I20260929 16:46:45.125622 20637 log.cc:824] T 40e0f419294943aba52f41c6faa1c45c P e4cbfb5e73004cc5867df1c6e836d5ee: Log is configured to *not* fsync() on all Append() calls
I20260929 16:46:45.128962 20636 tablet_bootstrap.cc:492] T 40e0f419294943aba52f41c6faa1c45c P c7ac916426934c799550e67f33615133: Bootstrap starting.
I20260929 16:46:45.133287 20635 tablet_bootstrap.cc:492] T 40e0f419294943aba52f41c6faa1c45c P f518e988785a4161b3e9c7a39effdf45: No bootstrap required, opened a new log
I20260929 16:46:45.133623 20636 tablet_bootstrap.cc:654] T 40e0f419294943aba52f41c6faa1c45c P c7ac916426934c799550e67f33615133: Neither blocks nor log segments found. Creating new log.
I20260929 16:46:45.133675 20635 ts_tablet_manager.cc:1397] T 40e0f419294943aba52f41c6faa1c45c P f518e988785a4161b3e9c7a39effdf45: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.011s
I20260929 16:46:45.135187 20636 log.cc:824] T 40e0f419294943aba52f41c6faa1c45c P c7ac916426934c799550e67f33615133: Log is configured to *not* fsync() on all Append() calls
I20260929 16:46:45.136593 20637 tablet_bootstrap.cc:492] T 40e0f419294943aba52f41c6faa1c45c P e4cbfb5e73004cc5867df1c6e836d5ee: No bootstrap required, opened a new log
I20260929 16:46:45.137027 20637 ts_tablet_manager.cc:1397] T 40e0f419294943aba52f41c6faa1c45c P e4cbfb5e73004cc5867df1c6e836d5ee: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.003s
I20260929 16:46:45.145219 20635 raft_consensus.cc:348] T 40e0f419294943aba52f41c6faa1c45c P f518e988785a4161b3e9c7a39effdf45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f518e988785a4161b3e9c7a39effdf45" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 37639 } } peers { permanent_uuid: "c7ac916426934c799550e67f33615133" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 42959 } } peers { permanent_uuid: "e4cbfb5e73004cc5867df1c6e836d5ee" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 36589 } }
I20260929 16:46:45.145793 20635 raft_consensus.cc:374] T 40e0f419294943aba52f41c6faa1c45c P f518e988785a4161b3e9c7a39effdf45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:46:45.145902 20635 raft_consensus.cc:729] T 40e0f419294943aba52f41c6faa1c45c P f518e988785a4161b3e9c7a39effdf45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f518e988785a4161b3e9c7a39effdf45, State: Initialized, Role: FOLLOWER
I20260929 16:46:45.146752 20635 consensus_queue.cc:260] T 40e0f419294943aba52f41c6faa1c45c P f518e988785a4161b3e9c7a39effdf45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f518e988785a4161b3e9c7a39effdf45" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 37639 } } peers { permanent_uuid: "c7ac916426934c799550e67f33615133" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 42959 } } peers { permanent_uuid: "e4cbfb5e73004cc5867df1c6e836d5ee" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 36589 } }
I20260929 16:46:45.150779 20637 raft_consensus.cc:348] T 40e0f419294943aba52f41c6faa1c45c P e4cbfb5e73004cc5867df1c6e836d5ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f518e988785a4161b3e9c7a39effdf45" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 37639 } } peers { permanent_uuid: "c7ac916426934c799550e67f33615133" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 42959 } } peers { permanent_uuid: "e4cbfb5e73004cc5867df1c6e836d5ee" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 36589 } }
I20260929 16:46:45.151614 20637 raft_consensus.cc:374] T 40e0f419294943aba52f41c6faa1c45c P e4cbfb5e73004cc5867df1c6e836d5ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:46:45.151746 20636 tablet_bootstrap.cc:492] T 40e0f419294943aba52f41c6faa1c45c P c7ac916426934c799550e67f33615133: No bootstrap required, opened a new log
I20260929 16:46:45.151813 20637 raft_consensus.cc:729] T 40e0f419294943aba52f41c6faa1c45c P e4cbfb5e73004cc5867df1c6e836d5ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cbfb5e73004cc5867df1c6e836d5ee, State: Initialized, Role: FOLLOWER
I20260929 16:46:45.152099 20636 ts_tablet_manager.cc:1397] T 40e0f419294943aba52f41c6faa1c45c P c7ac916426934c799550e67f33615133: Time spent bootstrapping tablet: real 0.023s user 0.018s sys 0.004s
I20260929 16:46:45.152840 20637 consensus_queue.cc:260] T 40e0f419294943aba52f41c6faa1c45c P e4cbfb5e73004cc5867df1c6e836d5ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f518e988785a4161b3e9c7a39effdf45" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 37639 } } peers { permanent_uuid: "c7ac916426934c799550e67f33615133" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 42959 } } peers { permanent_uuid: "e4cbfb5e73004cc5867df1c6e836d5ee" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 36589 } }
I20260929 16:46:45.155684 20635 ts_tablet_manager.cc:1428] T 40e0f419294943aba52f41c6faa1c45c P f518e988785a4161b3e9c7a39effdf45: Time spent starting tablet: real 0.022s user 0.013s sys 0.007s
I20260929 16:46:45.166733 20482 heartbeater.cc:503] Master 127.0.197.126:44915 was elected leader, sending a full tablet report...
I20260929 16:46:45.167678 20637 ts_tablet_manager.cc:1428] T 40e0f419294943aba52f41c6faa1c45c P e4cbfb5e73004cc5867df1c6e836d5ee: Time spent starting tablet: real 0.030s user 0.014s sys 0.014s
I20260929 16:46:45.168475 20636 raft_consensus.cc:348] T 40e0f419294943aba52f41c6faa1c45c P c7ac916426934c799550e67f33615133 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f518e988785a4161b3e9c7a39effdf45" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 37639 } } peers { permanent_uuid: "c7ac916426934c799550e67f33615133" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 42959 } } peers { permanent_uuid: "e4cbfb5e73004cc5867df1c6e836d5ee" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 36589 } }
I20260929 16:46:45.169127 20636 raft_consensus.cc:374] T 40e0f419294943aba52f41c6faa1c45c P c7ac916426934c799550e67f33615133 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:46:45.169250 20636 raft_consensus.cc:729] T 40e0f419294943aba52f41c6faa1c45c P c7ac916426934c799550e67f33615133 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7ac916426934c799550e67f33615133, State: Initialized, Role: FOLLOWER
I20260929 16:46:45.170127 20636 consensus_queue.cc:260] T 40e0f419294943aba52f41c6faa1c45c P c7ac916426934c799550e67f33615133 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f518e988785a4161b3e9c7a39effdf45" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 37639 } } peers { permanent_uuid: "c7ac916426934c799550e67f33615133" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 42959 } } peers { permanent_uuid: "e4cbfb5e73004cc5867df1c6e836d5ee" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 36589 } }
I20260929 16:46:45.171514 20616 heartbeater.cc:503] Master 127.0.197.126:44915 was elected leader, sending a full tablet report...
I20260929 16:46:45.174023 20636 ts_tablet_manager.cc:1428] T 40e0f419294943aba52f41c6faa1c45c P c7ac916426934c799550e67f33615133: Time spent starting tablet: real 0.022s user 0.018s sys 0.000s
W20260929 16:46:45.174291 20483 tablet.cc:2369] T 40e0f419294943aba52f41c6faa1c45c P e4cbfb5e73004cc5867df1c6e836d5ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:46:45.227588 20617 tablet.cc:2369] T 40e0f419294943aba52f41c6faa1c45c P c7ac916426934c799550e67f33615133: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:46:45.330498 20641 raft_consensus.cc:482] T 40e0f419294943aba52f41c6faa1c45c P f518e988785a4161b3e9c7a39effdf45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:46:45.330912 20641 raft_consensus.cc:504] T 40e0f419294943aba52f41c6faa1c45c P f518e988785a4161b3e9c7a39effdf45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f518e988785a4161b3e9c7a39effdf45" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 37639 } } peers { permanent_uuid: "c7ac916426934c799550e67f33615133" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 42959 } } peers { permanent_uuid: "e4cbfb5e73004cc5867df1c6e836d5ee" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 36589 } }
I20260929 16:46:45.333494 20641 leader_election.cc:288] T 40e0f419294943aba52f41c6faa1c45c P f518e988785a4161b3e9c7a39effdf45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7ac916426934c799550e67f33615133 (127.0.197.67:42959), e4cbfb5e73004cc5867df1c6e836d5ee (127.0.197.66:36589)
I20260929 16:46:45.343045 20632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:45.335332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:56113 (local address 127.0.197.66:36589)
0929 16:46:45.335618 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:45.335624 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:45.336630 (+ 1006us) server_negotiation.cc:407] Connection header received
0929 16:46:45.336907 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:45.336912 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:45.336985 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:45.337091 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:46:45.337715 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:45.338491 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:45.340824 (+ 2333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:45.341805 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:45.341980 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:45.342063 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:45.342535 (+ 472us) server_negotiation.cc:299] Negotiation successful
0929 16:46:45.342690 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260929 16:46:45.343402 20645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:45.334476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.66:36589 (local address 127.0.197.65:56113)
0929 16:46:45.336418 (+ 1942us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:45.336485 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0929 16:46:45.336719 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:46:45.337106 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:46:45.337111 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:46:45.337135 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:46:45.337524 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:45.337536 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:45.338645 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:45.338651 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:45.340619 (+ 1968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:45.340636 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:45.342056 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:45.342065 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:45.342299 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:45.342353 (+ 54us) client_negotiation.cc:769] Sending connection context
0929 16:46:45.342470 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 16:46:45.342677 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1794,"mutex_wait_us":1164,"thread_start_us":513,"threads_started":1}
I20260929 16:46:45.344362 20644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:45.334476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.67:42959 (local address 127.0.197.65:48415)
0929 16:46:45.335511 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:45.335547 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 16:46:45.335833 (+ 286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:46:45.336508 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:46:45.336516 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:46:45.336564 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:46:45.336996 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:45.337011 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:45.338704 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:45.338712 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:45.341424 (+ 2712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:45.341446 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:45.343377 (+ 1931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:45.343383 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:45.343627 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:45.343664 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 16:46:45.343794 (+ 130us) client_negotiation.cc:240] Negotiation successful
0929 16:46:45.343956 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":834,"mutex_wait_us":5,"thread_start_us":728,"threads_started":1}
I20260929 16:46:45.344795 20634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:45.334573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:48415 (local address 127.0.197.67:42959)
0929 16:46:45.334833 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:45.334841 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:45.335676 (+ 835us) server_negotiation.cc:407] Connection header received
0929 16:46:45.336021 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:45.336028 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:45.336153 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:45.336227 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:46:45.337190 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:45.338492 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:45.341642 (+ 3150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:45.342766 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:45.342962 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:45.343076 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:45.343827 (+ 751us) server_negotiation.cc:299] Negotiation successful
0929 16:46:45.343966 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 16:46:45.344767 20437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40e0f419294943aba52f41c6faa1c45c" candidate_uuid: "f518e988785a4161b3e9c7a39effdf45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cbfb5e73004cc5867df1c6e836d5ee" is_pre_election: true
I20260929 16:46:45.345463 20437 raft_consensus.cc:2454] T 40e0f419294943aba52f41c6faa1c45c P e4cbfb5e73004cc5867df1c6e836d5ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f518e988785a4161b3e9c7a39effdf45 in term 0.
I20260929 16:46:45.345790 20571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40e0f419294943aba52f41c6faa1c45c" candidate_uuid: "f518e988785a4161b3e9c7a39effdf45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7ac916426934c799550e67f33615133" is_pre_election: true
I20260929 16:46:45.346412 20238 leader_election.cc:302] T 40e0f419294943aba52f41c6faa1c45c P f518e988785a4161b3e9c7a39effdf45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e4cbfb5e73004cc5867df1c6e836d5ee, f518e988785a4161b3e9c7a39effdf45; no voters:
I20260929 16:46:45.346537 20571 raft_consensus.cc:2454] T 40e0f419294943aba52f41c6faa1c45c P c7ac916426934c799550e67f33615133 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f518e988785a4161b3e9c7a39effdf45 in term 0.
I20260929 16:46:45.346974 20641 raft_consensus.cc:2781] T 40e0f419294943aba52f41c6faa1c45c P f518e988785a4161b3e9c7a39effdf45 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:46:45.347185 20641 raft_consensus.cc:482] T 40e0f419294943aba52f41c6faa1c45c P f518e988785a4161b3e9c7a39effdf45 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:46:45.347303 20641 raft_consensus.cc:3037] T 40e0f419294943aba52f41c6faa1c45c P f518e988785a4161b3e9c7a39effdf45 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:46:45.351374 20641 raft_consensus.cc:504] T 40e0f419294943aba52f41c6faa1c45c P f518e988785a4161b3e9c7a39effdf45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f518e988785a4161b3e9c7a39effdf45" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 37639 } } peers { permanent_uuid: "c7ac916426934c799550e67f33615133" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 42959 } } peers { permanent_uuid: "e4cbfb5e73004cc5867df1c6e836d5ee" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 36589 } }
I20260929 16:46:45.352463 20641 leader_election.cc:288] T 40e0f419294943aba52f41c6faa1c45c P f518e988785a4161b3e9c7a39effdf45 [CANDIDATE]: Term 1 election: Requested vote from peers c7ac916426934c799550e67f33615133 (127.0.197.67:42959), e4cbfb5e73004cc5867df1c6e836d5ee (127.0.197.66:36589)
I20260929 16:46:45.353813 20571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40e0f419294943aba52f41c6faa1c45c" candidate_uuid: "f518e988785a4161b3e9c7a39effdf45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7ac916426934c799550e67f33615133"
I20260929 16:46:45.353945 20437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40e0f419294943aba52f41c6faa1c45c" candidate_uuid: "f518e988785a4161b3e9c7a39effdf45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4cbfb5e73004cc5867df1c6e836d5ee"
I20260929 16:46:45.354102 20571 raft_consensus.cc:3037] T 40e0f419294943aba52f41c6faa1c45c P c7ac916426934c799550e67f33615133 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:46:45.354225 20437 raft_consensus.cc:3037] T 40e0f419294943aba52f41c6faa1c45c P e4cbfb5e73004cc5867df1c6e836d5ee [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:46:45.357228 20437 raft_consensus.cc:2454] T 40e0f419294943aba52f41c6faa1c45c P e4cbfb5e73004cc5867df1c6e836d5ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f518e988785a4161b3e9c7a39effdf45 in term 1.
I20260929 16:46:45.357810 20238 leader_election.cc:302] T 40e0f419294943aba52f41c6faa1c45c P f518e988785a4161b3e9c7a39effdf45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e4cbfb5e73004cc5867df1c6e836d5ee, f518e988785a4161b3e9c7a39effdf45; no voters:
I20260929 16:46:45.358201 20641 raft_consensus.cc:2781] T 40e0f419294943aba52f41c6faa1c45c P f518e988785a4161b3e9c7a39effdf45 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:46:45.358197 20571 raft_consensus.cc:2454] T 40e0f419294943aba52f41c6faa1c45c P c7ac916426934c799550e67f33615133 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f518e988785a4161b3e9c7a39effdf45 in term 1.
I20260929 16:46:45.358817 20641 raft_consensus.cc:686] T 40e0f419294943aba52f41c6faa1c45c P f518e988785a4161b3e9c7a39effdf45 [term 1 LEADER]: Becoming Leader. State: Replica: f518e988785a4161b3e9c7a39effdf45, State: Running, Role: LEADER
I20260929 16:46:45.359467 20641 consensus_queue.cc:237] T 40e0f419294943aba52f41c6faa1c45c P f518e988785a4161b3e9c7a39effdf45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f518e988785a4161b3e9c7a39effdf45" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 37639 } } peers { permanent_uuid: "c7ac916426934c799550e67f33615133" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 42959 } } peers { permanent_uuid: "e4cbfb5e73004cc5867df1c6e836d5ee" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 36589 } }
I20260929 16:46:45.365486 20158 catalog_manager.cc:5495] T 40e0f419294943aba52f41c6faa1c45c P f518e988785a4161b3e9c7a39effdf45 reported cstate change: term changed from 0 to 1, leader changed from <none> to f518e988785a4161b3e9c7a39effdf45 (127.0.197.65). New cstate: current_term: 1 leader_uuid: "f518e988785a4161b3e9c7a39effdf45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f518e988785a4161b3e9c7a39effdf45" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 37639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c7ac916426934c799550e67f33615133" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 42959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4cbfb5e73004cc5867df1c6e836d5ee" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 36589 } health_report { overall_health: UNKNOWN } } }
W20260929 16:46:45.367074 20349 tablet.cc:2369] T 40e0f419294943aba52f41c6faa1c45c P f518e988785a4161b3e9c7a39effdf45: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:46:45.450919 20630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:45.444794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47014 (local address 127.0.197.65:37639)
0929 16:46:45.445077 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:45.445084 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:45.445107 (+ 23us) server_negotiation.cc:407] Connection header received
0929 16:46:45.445351 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:45.445355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:45.445449 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:45.445640 (+ 191us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:46:45.446043 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:45.446842 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:45.448423 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:45.448886 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:45.449054 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:45.449639 (+ 585us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:46:45.450222 (+ 583us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:46:45.450409 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:45.450505 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 16:46:45.450605 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 16:46:45.460469 20437 raft_consensus.cc:1261] T 40e0f419294943aba52f41c6faa1c45c P e4cbfb5e73004cc5867df1c6e836d5ee [term 1 FOLLOWER]: Refusing update from remote peer f518e988785a4161b3e9c7a39effdf45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:46:45.460471 20571 raft_consensus.cc:1261] T 40e0f419294943aba52f41c6faa1c45c P c7ac916426934c799550e67f33615133 [term 1 FOLLOWER]: Refusing update from remote peer f518e988785a4161b3e9c7a39effdf45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:46:45.461647 20646 consensus_queue.cc:1035] T 40e0f419294943aba52f41c6faa1c45c P f518e988785a4161b3e9c7a39effdf45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7ac916426934c799550e67f33615133" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 42959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:46:45.462059 20641 consensus_queue.cc:1035] T 40e0f419294943aba52f41c6faa1c45c P f518e988785a4161b3e9c7a39effdf45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4cbfb5e73004cc5867df1c6e836d5ee" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 36589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:46:45.475078 20649 mvcc.cc:205] Tried to move back new op lower bound from 7334708860754874368 to 7334708860357001216. Current Snapshot: MvccSnapshot[applied={T|T < 7334708860754874368}]
I20260929 16:46:45.495631 20158 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55052:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:46:45.495931 20158 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55052:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:46:45.500662 20158 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a961cd3f91149b7af4fe109223c2365: Sending DeleteTablet for 3 replicas of tablet 40e0f419294943aba52f41c6faa1c45c
I20260929 16:46:45.502167 20283 tablet_service.cc:1514] Processing DeleteTablet for tablet 40e0f419294943aba52f41c6faa1c45c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:46:45 UTC) from {username='kudu', principal='kudu/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:47008
I20260929 16:46:45.502565 20551 tablet_service.cc:1514] Processing DeleteTablet for tablet 40e0f419294943aba52f41c6faa1c45c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:46:45 UTC) from {username='kudu', principal='kudu/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:50996
I20260929 16:46:45.502482 20550 tablet_service.cc:1514] Processing DeleteTablet for tablet 40e0f419294943aba52f41c6faa1c45c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:46:45 UTC) from {username='kudu', principal='kudu/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:50996
I20260929 16:46:45.503659 20659 tablet_replica.cc:330] T 40e0f419294943aba52f41c6faa1c45c P c7ac916426934c799550e67f33615133: stopping tablet replica
I20260929 16:46:45.503507 20417 tablet_service.cc:1514] Processing DeleteTablet for tablet 40e0f419294943aba52f41c6faa1c45c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:46:45 UTC) from {username='kudu', principal='kudu/127.0.197.126@KRBTEST.COM'} at 127.0.0.1:50154
I20260929 16:46:45.504209 20659 raft_consensus.cc:2229] T 40e0f419294943aba52f41c6faa1c45c P c7ac916426934c799550e67f33615133 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:46:45.504335 20660 tablet_replica.cc:330] T 40e0f419294943aba52f41c6faa1c45c P f518e988785a4161b3e9c7a39effdf45: stopping tablet replica
I20260929 16:46:45.504873 20659 raft_consensus.cc:2258] T 40e0f419294943aba52f41c6faa1c45c P c7ac916426934c799550e67f33615133 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:46:45.505321 20660 raft_consensus.cc:2229] T 40e0f419294943aba52f41c6faa1c45c P f518e988785a4161b3e9c7a39effdf45 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:46:45.505897 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 20220
I20260929 16:46:45.506711 20660 raft_consensus.cc:2258] T 40e0f419294943aba52f41c6faa1c45c P f518e988785a4161b3e9c7a39effdf45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:46:45.508574 20659 ts_tablet_manager.cc:1905] T 40e0f419294943aba52f41c6faa1c45c P c7ac916426934c799550e67f33615133: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:46:45.515693 20661 tablet_replica.cc:330] T 40e0f419294943aba52f41c6faa1c45c P e4cbfb5e73004cc5867df1c6e836d5ee: stopping tablet replica
I20260929 16:46:45.516326 20661 raft_consensus.cc:2229] T 40e0f419294943aba52f41c6faa1c45c P e4cbfb5e73004cc5867df1c6e836d5ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:46:45.516999 20661 raft_consensus.cc:2258] T 40e0f419294943aba52f41c6faa1c45c P e4cbfb5e73004cc5867df1c6e836d5ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:46:45.518417 20659 ts_tablet_manager.cc:1918] T 40e0f419294943aba52f41c6faa1c45c P c7ac916426934c799550e67f33615133: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:46:45.518540 20660 ts_tablet_manager.cc:1905] T 40e0f419294943aba52f41c6faa1c45c P f518e988785a4161b3e9c7a39effdf45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:46:45.518648 20659 log.cc:1196] T 40e0f419294943aba52f41c6faa1c45c P c7ac916426934c799550e67f33615133: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/ts-2/wal/wals/40e0f419294943aba52f41c6faa1c45c
I20260929 16:46:45.519285 20659 ts_tablet_manager.cc:1939] T 40e0f419294943aba52f41c6faa1c45c P c7ac916426934c799550e67f33615133: Deleting consensus metadata
I20260929 16:46:45.521775 20146 catalog_manager.cc:4841] TS c7ac916426934c799550e67f33615133 (127.0.197.67:42959): tablet 40e0f419294943aba52f41c6faa1c45c (table test-table [id=571330147bd840a8af10589999300703]) successfully deleted
W20260929 16:46:45.522977 20146 catalog_manager.cc:4808] TS c7ac916426934c799550e67f33615133 (127.0.197.67:42959): delete failed for tablet 40e0f419294943aba52f41c6faa1c45c because the tablet was not found. No further retry: Not found: Tablet not found: 40e0f419294943aba52f41c6faa1c45c
I20260929 16:46:45.524602 20661 ts_tablet_manager.cc:1905] T 40e0f419294943aba52f41c6faa1c45c P e4cbfb5e73004cc5867df1c6e836d5ee: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 16:46:45 dist-test-slave-l31f krb5kdc[20114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.65@KRBTEST.COM
I20260929 16:46:45.534216 20661 ts_tablet_manager.cc:1918] T 40e0f419294943aba52f41c6faa1c45c P e4cbfb5e73004cc5867df1c6e836d5ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:46:45.534438 20661 log.cc:1196] T 40e0f419294943aba52f41c6faa1c45c P e4cbfb5e73004cc5867df1c6e836d5ee: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/ts-1/wal/wals/40e0f419294943aba52f41c6faa1c45c
I20260929 16:46:45.534833 20660 ts_tablet_manager.cc:1918] T 40e0f419294943aba52f41c6faa1c45c P f518e988785a4161b3e9c7a39effdf45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:46:45.534994 20661 ts_tablet_manager.cc:1939] T 40e0f419294943aba52f41c6faa1c45c P e4cbfb5e73004cc5867df1c6e836d5ee: Deleting consensus metadata
I20260929 16:46:45.535099 20660 log.cc:1196] T 40e0f419294943aba52f41c6faa1c45c P f518e988785a4161b3e9c7a39effdf45: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790700068255611-789-0/minicluster-data/ts-0/wal/wals/40e0f419294943aba52f41c6faa1c45c
I20260929 16:46:45.535693 20660 ts_tablet_manager.cc:1939] T 40e0f419294943aba52f41c6faa1c45c P f518e988785a4161b3e9c7a39effdf45: Deleting consensus metadata
I20260929 16:46:45.537386 20145 catalog_manager.cc:4841] TS e4cbfb5e73004cc5867df1c6e836d5ee (127.0.197.66:36589): tablet 40e0f419294943aba52f41c6faa1c45c (table test-table [id=571330147bd840a8af10589999300703]) successfully deleted
I20260929 16:46:45.537758 20146 catalog_manager.cc:4841] TS f518e988785a4161b3e9c7a39effdf45 (127.0.197.65:37639): tablet 40e0f419294943aba52f41c6faa1c45c (table test-table [id=571330147bd840a8af10589999300703]) successfully deleted
I20260929 16:46:45.538877 20630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:45.507350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47024 (local address 127.0.197.65:37639)
0929 16:46:45.507609 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:45.507616 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:45.515961 (+ 8345us) server_negotiation.cc:407] Connection header received
0929 16:46:45.516305 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:45.516311 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:45.516409 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:45.516488 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:45.518465 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:45.519787 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:45.521745 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:45.522578 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:45.522697 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:45.528903 (+ 6206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:45.528921 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:45.528937 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:46:45.528985 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:45.533937 (+ 4952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:45.535600 (+ 1663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:45.535611 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:45.535614 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:45.535914 (+ 300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:45.536453 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:45.536458 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:45.536461 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:45.537043 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:45.537625 (+ 582us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:45.538237 (+ 612us) server_negotiation.cc:299] Negotiation successful
0929 16:46:45.538508 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 16:46:45.539666 20343 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:46:45.639415 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 20220
I20260929 16:46:45.658715 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 20354
Sep 29 16:46:45 dist-test-slave-l31f krb5kdc[20114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.66@KRBTEST.COM
I20260929 16:46:45.675442 20632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:45.659673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50162 (local address 127.0.197.66:36589)
0929 16:46:45.659976 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:45.659982 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:45.660002 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:46:45.660217 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:45.660223 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:45.660300 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:45.660412 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:45.661685 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:45.662429 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:45.663864 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:45.664282 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:45.664425 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:45.669167 (+ 4742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:45.669184 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:45.669201 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:46:45.669241 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:45.672372 (+ 3131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:45.673223 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:45.673231 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:45.673233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:45.673369 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:45.673752 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:45.673757 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:45.673760 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:45.674175 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:45.674534 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:45.674798 (+ 264us) server_negotiation.cc:299] Negotiation successful
0929 16:46:45.675059 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169}
I20260929 16:46:45.676156 20477 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:46:45.772773 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 20354
I20260929 16:46:45.790573 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 20488
Sep 29 16:46:45 dist-test-slave-l31f krb5kdc[20114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.67@KRBTEST.COM
I20260929 16:46:45.809252 20634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:45.791530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51004 (local address 127.0.197.67:42959)
0929 16:46:45.791724 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:45.791729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:45.791748 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:46:45.791808 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:45.791812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:45.791907 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:45.792003 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:45.793523 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:45.794427 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:45.796009 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:45.796588 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:45.796710 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:45.801829 (+ 5119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:45.801856 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:45.801877 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:46:45.801935 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:45.805659 (+ 3724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:45.806630 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:45.806638 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:45.806640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:45.806825 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:45.807550 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:45.807555 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:45.807563 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:45.808010 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:45.808279 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:45.808752 (+ 473us) server_negotiation.cc:299] Negotiation successful
0929 16:46:45.808976 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 16:46:45.810514 20611 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:46:45.904537 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 20488
I20260929 16:46:45.922277 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 20127
I20260929 16:46:45.923732 20189 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:46:46.037730 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 20127
2026-09-29T16:46:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4755 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-29T16:46:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:46:46Z Disabled control of system clock
I20260929 16:46:46.121393 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:46209
--webserver_interface=127.0.197.126
--webserver_port=0
--builtin_ntp_servers=127.0.197.84:37263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.126:46209
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 16:46:46.478511 20670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:46:46.478843 20670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:46:46.479065 20670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:46:46.486831 20670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:46:46.486984 20670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:46:46.487106 20670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:46:46.487205 20670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:46:46.487296 20670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:46:46.497474 20670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:37263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.197.126:46209
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:46209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.197.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:46:46.498926 20670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:46:46.500909 20670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:46:46.513932 20678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:46.514073 20676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:46.514073 20675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:46:46.515311 20670 server_base.cc:1034] running on GCE node
I20260929 16:46:46.516204 20670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:46:46.517907 20670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:46:46.519133 20670 hybrid_clock.cc:648] HybridClock initialized: now 1790700406519079 us; error 49 us; skew 500 ppm
I20260929 16:46:46.519539 20670 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:46:46.530413 20670 webserver.cc:460] Webserver started at http://127.0.197.126:38875/ using document root <none> and password file <none>
I20260929 16:46:46.531414 20670 fs_manager.cc:359] Metadata directory not provided
I20260929 16:46:46.531575 20670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:46:46.531953 20670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:46:46.535272 20670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/master-0/data/instance:
uuid: "f66f7d6a8d194b1c895598c8f2cef207"
format_stamp: "Formatted at 2026-09-29 16:46:46 on dist-test-slave-l31f"
server_key: "17c496ac6aeaaf2b478e9dc794d0d305"
server_key_iv: "3f8c077bd6a8abb9a5966b9e4618c078"
server_key_version: "encryptionkey@0"
I20260929 16:46:46.536207 20670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/master-0/wal/instance:
uuid: "f66f7d6a8d194b1c895598c8f2cef207"
format_stamp: "Formatted at 2026-09-29 16:46:46 on dist-test-slave-l31f"
server_key: "17c496ac6aeaaf2b478e9dc794d0d305"
server_key_iv: "3f8c077bd6a8abb9a5966b9e4618c078"
server_key_version: "encryptionkey@0"
I20260929 16:46:46.542551 20670 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 16:46:46.548936 20684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:46.550949 20670 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.001s
I20260929 16:46:46.551223 20670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "f66f7d6a8d194b1c895598c8f2cef207"
format_stamp: "Formatted at 2026-09-29 16:46:46 on dist-test-slave-l31f"
server_key: "17c496ac6aeaaf2b478e9dc794d0d305"
server_key_iv: "3f8c077bd6a8abb9a5966b9e4618c078"
server_key_version: "encryptionkey@0"
I20260929 16:46:46.551509 20670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:46:46.590698 20670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:46:46.591703 20670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:46:46.592043 20670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:46:46.612363 20670 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.126:46209
I20260929 16:46:46.612385 20735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.126:46209 every 8 connection(s)
I20260929 16:46:46.613886 20670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:46:46.618068 20736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:46:46.620540 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 20670
I20260929 16:46:46.620728 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/master-0/wal/instance
I20260929 16:46:46.621286 789 external_mini_cluster.cc:1428] Setting key 3deebc8640c085016da4b7edbefaf92f
I20260929 16:46:46.631247 20736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f66f7d6a8d194b1c895598c8f2cef207: Bootstrap starting.
I20260929 16:46:46.634881 20736 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f66f7d6a8d194b1c895598c8f2cef207: Neither blocks nor log segments found. Creating new log.
I20260929 16:46:46.635918 20739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:46.623475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40886 (local address 127.0.197.126:46209)
0929 16:46:46.624589 (+ 1114us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:46.624602 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:46.624649 (+ 47us) server_negotiation.cc:407] Connection header received
0929 16:46:46.626470 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:46.626498 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:46.627144 (+ 646us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:46.627576 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:46.628246 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:46.629625 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:46.631065 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:46.631914 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:46.632137 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:46.633433 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:46.633442 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:46.633462 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:46:46.634022 (+ 560us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:46.634109 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:46:46.634131 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:46.634267 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:46.634497 (+ 230us) server_negotiation.cc:299] Negotiation successful
0929 16:46:46.634834 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":404,"threads_started":1}
I20260929 16:46:46.636087 20736 log.cc:824] T 00000000000000000000000000000000 P f66f7d6a8d194b1c895598c8f2cef207: Log is configured to *not* fsync() on all Append() calls
I20260929 16:46:46.639285 20736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f66f7d6a8d194b1c895598c8f2cef207: No bootstrap required, opened a new log
I20260929 16:46:46.647285 20736 raft_consensus.cc:348] T 00000000000000000000000000000000 P f66f7d6a8d194b1c895598c8f2cef207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f66f7d6a8d194b1c895598c8f2cef207" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 46209 } }
I20260929 16:46:46.647703 20736 raft_consensus.cc:374] T 00000000000000000000000000000000 P f66f7d6a8d194b1c895598c8f2cef207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:46:46.647785 20736 raft_consensus.cc:729] T 00000000000000000000000000000000 P f66f7d6a8d194b1c895598c8f2cef207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f66f7d6a8d194b1c895598c8f2cef207, State: Initialized, Role: FOLLOWER
I20260929 16:46:46.648454 20736 consensus_queue.cc:260] T 00000000000000000000000000000000 P f66f7d6a8d194b1c895598c8f2cef207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f66f7d6a8d194b1c895598c8f2cef207" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 46209 } }
I20260929 16:46:46.648681 20736 raft_consensus.cc:388] T 00000000000000000000000000000000 P f66f7d6a8d194b1c895598c8f2cef207 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:46:46.648777 20736 raft_consensus.cc:482] T 00000000000000000000000000000000 P f66f7d6a8d194b1c895598c8f2cef207 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:46:46.648906 20736 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f66f7d6a8d194b1c895598c8f2cef207 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:46:46.651650 20736 raft_consensus.cc:504] T 00000000000000000000000000000000 P f66f7d6a8d194b1c895598c8f2cef207 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f66f7d6a8d194b1c895598c8f2cef207" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 46209 } }
I20260929 16:46:46.652184 20736 leader_election.cc:302] T 00000000000000000000000000000000 P f66f7d6a8d194b1c895598c8f2cef207 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f66f7d6a8d194b1c895598c8f2cef207; no voters:
I20260929 16:46:46.652768 20736 leader_election.cc:288] T 00000000000000000000000000000000 P f66f7d6a8d194b1c895598c8f2cef207 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:46:46.652941 20741 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f66f7d6a8d194b1c895598c8f2cef207 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:46:46.653693 20741 raft_consensus.cc:686] T 00000000000000000000000000000000 P f66f7d6a8d194b1c895598c8f2cef207 [term 1 LEADER]: Becoming Leader. State: Replica: f66f7d6a8d194b1c895598c8f2cef207, State: Running, Role: LEADER
I20260929 16:46:46.654239 20741 consensus_queue.cc:237] T 00000000000000000000000000000000 P f66f7d6a8d194b1c895598c8f2cef207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f66f7d6a8d194b1c895598c8f2cef207" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 46209 } }
I20260929 16:46:46.655092 20736 sys_catalog.cc:564] T 00000000000000000000000000000000 P f66f7d6a8d194b1c895598c8f2cef207 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:46:46.657145 20743 sys_catalog.cc:455] T 00000000000000000000000000000000 P f66f7d6a8d194b1c895598c8f2cef207 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f66f7d6a8d194b1c895598c8f2cef207. Latest consensus state: current_term: 1 leader_uuid: "f66f7d6a8d194b1c895598c8f2cef207" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f66f7d6a8d194b1c895598c8f2cef207" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 46209 } } }
I20260929 16:46:46.657588 20743 sys_catalog.cc:458] T 00000000000000000000000000000000 P f66f7d6a8d194b1c895598c8f2cef207 [sys.catalog]: This master's current role is: LEADER
I20260929 16:46:46.658196 20742 sys_catalog.cc:455] T 00000000000000000000000000000000 P f66f7d6a8d194b1c895598c8f2cef207 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f66f7d6a8d194b1c895598c8f2cef207" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f66f7d6a8d194b1c895598c8f2cef207" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 46209 } } }
I20260929 16:46:46.658599 20742 sys_catalog.cc:458] T 00000000000000000000000000000000 P f66f7d6a8d194b1c895598c8f2cef207 [sys.catalog]: This master's current role is: LEADER
I20260929 16:46:46.661190 20746 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:46:46.666841 20746 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:46:46.677562 20746 catalog_manager.cc:1347] Generated new cluster ID: 8aca0573218c4f54b18b01ae35f3bfcb
I20260929 16:46:46.677698 20746 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:46:46.690210 20746 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:46:46.692178 20746 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:46:46.703217 20746 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f66f7d6a8d194b1c895598c8f2cef207: Generated new TSK 0
I20260929 16:46:46.704429 20746 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:46:46.715649 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.65:0
--local_ip_for_outbound_sockets=127.0.197.65
--webserver_interface=127.0.197.65
--webserver_port=0
--tserver_master_addrs=127.0.197.126:46209
--builtin_ntp_servers=127.0.197.84:37263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 16:46:47.083462 20760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:46:47.083773 20760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:46:47.083915 20760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:46:47.091741 20760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:46:47.091866 20760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:46:47.092070 20760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.65
I20260929 16:46:47.102481 20760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:37263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.197.65
--webserver_port=0
--tserver_master_addrs=127.0.197.126:46209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.65
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:46:47.104008 20760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:46:47.105995 20760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:46:47.121259 20765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:47.121333 20766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:47.121346 20768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:46:47.133064 20760 server_base.cc:1034] running on GCE node
I20260929 16:46:47.133847 20760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:46:47.135242 20760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:46:47.136514 20760 hybrid_clock.cc:648] HybridClock initialized: now 1790700407136442 us; error 62 us; skew 500 ppm
I20260929 16:46:47.136910 20760 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:46:47.139765 20760 webserver.cc:460] Webserver started at http://127.0.197.65:46463/ using document root <none> and password file <none>
I20260929 16:46:47.140578 20760 fs_manager.cc:359] Metadata directory not provided
I20260929 16:46:47.140683 20760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:46:47.141115 20760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:46:47.144423 20760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/ts-0/data/instance:
uuid: "025e7e81bd6c41b4965c08cf8e94b79e"
format_stamp: "Formatted at 2026-09-29 16:46:47 on dist-test-slave-l31f"
server_key: "200bbbdd4bb9628bbe5b435d35d4daa7"
server_key_iv: "1e435c99b71d00e25626d5357044eb62"
server_key_version: "encryptionkey@0"
I20260929 16:46:47.145344 20760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/ts-0/wal/instance:
uuid: "025e7e81bd6c41b4965c08cf8e94b79e"
format_stamp: "Formatted at 2026-09-29 16:46:47 on dist-test-slave-l31f"
server_key: "200bbbdd4bb9628bbe5b435d35d4daa7"
server_key_iv: "1e435c99b71d00e25626d5357044eb62"
server_key_version: "encryptionkey@0"
I20260929 16:46:47.151185 20760 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 16:46:47.155467 20774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:47.157389 20760 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:46:47.157562 20760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/ts-0/wal
uuid: "025e7e81bd6c41b4965c08cf8e94b79e"
format_stamp: "Formatted at 2026-09-29 16:46:47 on dist-test-slave-l31f"
server_key: "200bbbdd4bb9628bbe5b435d35d4daa7"
server_key_iv: "1e435c99b71d00e25626d5357044eb62"
server_key_version: "encryptionkey@0"
I20260929 16:46:47.157773 20760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:46:47.173270 20760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:46:47.174127 20760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:46:47.174401 20760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:46:47.175547 20760 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:46:47.177143 20760 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:46:47.177273 20760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:47.177420 20760 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:46:47.177511 20760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:47.214679 20760 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.65:37355
I20260929 16:46:47.214774 20886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.65:37355 every 8 connection(s)
I20260929 16:46:47.216238 20760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/ts-0/data/info.pb
I20260929 16:46:47.224046 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 20760
I20260929 16:46:47.224299 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/ts-0/wal/instance
I20260929 16:46:47.224893 789 external_mini_cluster.cc:1428] Setting key 0a2191f7619348a1947169771ffef08d
I20260929 16:46:47.227988 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.66:0
--local_ip_for_outbound_sockets=127.0.197.66
--webserver_interface=127.0.197.66
--webserver_port=0
--tserver_master_addrs=127.0.197.126:46209
--builtin_ntp_servers=127.0.197.84:37263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 16:46:47.231307 20889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:47.219132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.65:46279 (local address 127.0.197.126:46209)
0929 16:46:47.219619 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:47.219624 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:47.220849 (+ 1225us) server_negotiation.cc:407] Connection header received
0929 16:46:47.222885 (+ 2036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:47.222891 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:47.223051 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:47.223184 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:47.224526 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:47.225618 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:47.227690 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:47.228407 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:47.228565 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:47.230192 (+ 1627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:47.230202 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:47.230209 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:46:47.230239 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:47.230297 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:46:47.230316 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:47.230418 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:47.230706 (+ 288us) server_negotiation.cc:299] Negotiation successful
0929 16:46:47.230835 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":261,"threads_started":1}
I20260929 16:46:47.232347 20890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:47.219667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:46209 (local address 127.0.197.65:46279)
0929 16:46:47.220664 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:47.220711 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 16:46:47.222537 (+ 1826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:46:47.223472 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:46:47.223481 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:46:47.223583 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:46:47.224269 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:47.224298 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:47.225835 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:47.225840 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:47.227506 (+ 1666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:47.227523 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:47.228718 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:47.228724 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:47.228972 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:47.229770 (+ 798us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:46:47.229773 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:46:47.229819 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:46:47.229945 (+ 126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:46:47.230456 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:46:47.230467 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:46:47.230474 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 16:46:47.230656 (+ 182us) client_negotiation.cc:240] Negotiation successful
0929 16:46:47.231123 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":307,"threads_started":1}
I20260929 16:46:47.234472 20887 heartbeater.cc:347] Connected to a master server at 127.0.197.126:46209
I20260929 16:46:47.234864 20887 heartbeater.cc:464] Registering TS with master...
I20260929 16:46:47.235941 20887 heartbeater.cc:511] Master 127.0.197.126:46209 requested a full tablet report, sending...
I20260929 16:46:47.238749 20701 ts_manager.cc:194] Registered new tserver with Master: 025e7e81bd6c41b4965c08cf8e94b79e (127.0.197.65:37355)
I20260929 16:46:47.240365 20701 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.197.65:46279
W20260929 16:46:47.611222 20891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:46:47.611636 20891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:46:47.611778 20891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:46:47.624464 20891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:46:47.624641 20891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:46:47.624812 20891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.66
I20260929 16:46:47.639245 20891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:37263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.197.66
--webserver_port=0
--tserver_master_addrs=127.0.197.126:46209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.66
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:46:47.640568 20891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:46:47.642396 20891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:46:47.657054 20896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:47.657084 20899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:47.657054 20897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:46:47.658612 20891 server_base.cc:1034] running on GCE node
I20260929 16:46:47.659467 20891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:46:47.660621 20891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:46:47.661899 20891 hybrid_clock.cc:648] HybridClock initialized: now 1790700407661836 us; error 59 us; skew 500 ppm
I20260929 16:46:47.662319 20891 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:46:47.665247 20891 webserver.cc:460] Webserver started at http://127.0.197.66:46461/ using document root <none> and password file <none>
I20260929 16:46:47.666059 20891 fs_manager.cc:359] Metadata directory not provided
I20260929 16:46:47.666204 20891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:46:47.666551 20891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:46:47.669697 20891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/ts-1/data/instance:
uuid: "4e811b45a0d74c838cfc5bff0a0fc77b"
format_stamp: "Formatted at 2026-09-29 16:46:47 on dist-test-slave-l31f"
server_key: "4a593c914677f4f11edae7a6589de97f"
server_key_iv: "a080e88fb001802ecdb0c827c6e37bf1"
server_key_version: "encryptionkey@0"
I20260929 16:46:47.670601 20891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/ts-1/wal/instance:
uuid: "4e811b45a0d74c838cfc5bff0a0fc77b"
format_stamp: "Formatted at 2026-09-29 16:46:47 on dist-test-slave-l31f"
server_key: "4a593c914677f4f11edae7a6589de97f"
server_key_iv: "a080e88fb001802ecdb0c827c6e37bf1"
server_key_version: "encryptionkey@0"
I20260929 16:46:47.676471 20891 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260929 16:46:47.680586 20905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:47.682158 20891 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:46:47.682397 20891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/ts-1/wal
uuid: "4e811b45a0d74c838cfc5bff0a0fc77b"
format_stamp: "Formatted at 2026-09-29 16:46:47 on dist-test-slave-l31f"
server_key: "4a593c914677f4f11edae7a6589de97f"
server_key_iv: "a080e88fb001802ecdb0c827c6e37bf1"
server_key_version: "encryptionkey@0"
I20260929 16:46:47.682677 20891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:46:47.705174 20891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:46:47.706174 20891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:46:47.706535 20891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:46:47.707720 20891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:46:47.709887 20891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:46:47.710034 20891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:47.710176 20891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:46:47.710258 20891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:47.745723 20891 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.66:40849
I20260929 16:46:47.745843 21017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.66:40849 every 8 connection(s)
I20260929 16:46:47.747213 20891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/ts-1/data/info.pb
I20260929 16:46:47.751884 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 20891
I20260929 16:46:47.752075 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/ts-1/wal/instance
I20260929 16:46:47.752636 789 external_mini_cluster.cc:1428] Setting key 607316bb6c5ddedb34f0cd8c72b7c355
I20260929 16:46:47.756343 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.197.67:0
--local_ip_for_outbound_sockets=127.0.197.67
--webserver_interface=127.0.197.67
--webserver_port=0
--tserver_master_addrs=127.0.197.126:46209
--builtin_ntp_servers=127.0.197.84:37263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 16:46:47.762837 21020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:47.749909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.66:35055 (local address 127.0.197.126:46209)
0929 16:46:47.750408 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:47.750415 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:47.751908 (+ 1493us) server_negotiation.cc:407] Connection header received
0929 16:46:47.754311 (+ 2403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:47.754315 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:47.754405 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:47.754506 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:47.756142 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:47.757349 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:47.759228 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:47.759973 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:47.760136 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:47.761814 (+ 1678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:47.761820 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:47.761823 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:46:47.761845 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:47.761887 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:46:47.761954 (+ 67us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:47.762046 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:47.762308 (+ 262us) server_negotiation.cc:299] Negotiation successful
0929 16:46:47.762452 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":246,"threads_started":1}
I20260929 16:46:47.764151 21021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:47.750570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:46209 (local address 127.0.197.66:35055)
0929 16:46:47.751702 (+ 1132us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:47.751759 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 16:46:47.753935 (+ 2176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:46:47.754805 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:46:47.754814 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:46:47.754963 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:46:47.755894 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:47.755925 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:47.757549 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:47.757553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:47.758969 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:47.758986 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:47.760254 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:47.760260 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:47.760504 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:47.761404 (+ 900us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:46:47.761407 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:46:47.761458 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:46:47.761567 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:46:47.762044 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:46:47.762056 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:46:47.762067 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 16:46:47.762278 (+ 211us) client_negotiation.cc:240] Negotiation successful
0929 16:46:47.762760 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":291,"threads_started":1}
I20260929 16:46:47.765981 21018 heartbeater.cc:347] Connected to a master server at 127.0.197.126:46209
I20260929 16:46:47.766352 21018 heartbeater.cc:464] Registering TS with master...
I20260929 16:46:47.767382 21018 heartbeater.cc:511] Master 127.0.197.126:46209 requested a full tablet report, sending...
I20260929 16:46:47.769184 20701 ts_manager.cc:194] Registered new tserver with Master: 4e811b45a0d74c838cfc5bff0a0fc77b (127.0.197.66:40849)
I20260929 16:46:47.770450 20701 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.197.66:35055
W20260929 16:46:48.130740 21022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:46:48.131172 21022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:46:48.131284 21022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:46:48.139468 21022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:46:48.139609 21022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:46:48.139726 21022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.197.67
I20260929 16:46:48.149641 21022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:37263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.197.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.197.67
--webserver_port=0
--tserver_master_addrs=127.0.197.126:46209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.197.67
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:46:48.150957 21022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:46:48.152817 21022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:46:48.168279 21027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:48.168226 21028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:48.168201 21030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:46:48.169071 21022 server_base.cc:1034] running on GCE node
I20260929 16:46:48.169771 21022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:46:48.170871 21022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:46:48.172128 21022 hybrid_clock.cc:648] HybridClock initialized: now 1790700408172076 us; error 51 us; skew 500 ppm
I20260929 16:46:48.172487 21022 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:46:48.175218 21022 webserver.cc:460] Webserver started at http://127.0.197.67:44427/ using document root <none> and password file <none>
I20260929 16:46:48.175984 21022 fs_manager.cc:359] Metadata directory not provided
I20260929 16:46:48.176127 21022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:46:48.176512 21022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:46:48.179677 21022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/ts-2/data/instance:
uuid: "3b2bb7d9b01740229bbb67268906df79"
format_stamp: "Formatted at 2026-09-29 16:46:48 on dist-test-slave-l31f"
server_key: "479c46ba78356842f7ff111224f95803"
server_key_iv: "77e22bf0313248076473fa5115d3f494"
server_key_version: "encryptionkey@0"
I20260929 16:46:48.180558 21022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/ts-2/wal/instance:
uuid: "3b2bb7d9b01740229bbb67268906df79"
format_stamp: "Formatted at 2026-09-29 16:46:48 on dist-test-slave-l31f"
server_key: "479c46ba78356842f7ff111224f95803"
server_key_iv: "77e22bf0313248076473fa5115d3f494"
server_key_version: "encryptionkey@0"
I20260929 16:46:48.186856 21022 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 16:46:48.191186 21036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:48.192849 21022 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260929 16:46:48.193092 21022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/ts-2/wal
uuid: "3b2bb7d9b01740229bbb67268906df79"
format_stamp: "Formatted at 2026-09-29 16:46:48 on dist-test-slave-l31f"
server_key: "479c46ba78356842f7ff111224f95803"
server_key_iv: "77e22bf0313248076473fa5115d3f494"
server_key_version: "encryptionkey@0"
I20260929 16:46:48.193367 21022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:46:48.222842 21022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:46:48.223893 21022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:46:48.224247 21022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:46:48.225333 21022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:46:48.226949 21022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:46:48.227114 21022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260929 16:46:48.227258 21022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:46:48.227351 21022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:48.244943 20887 heartbeater.cc:503] Master 127.0.197.126:46209 was elected leader, sending a full tablet report...
I20260929 16:46:48.266943 21022 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.67:37691
I20260929 16:46:48.267083 21148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.67:37691 every 8 connection(s)
I20260929 16:46:48.269124 21022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/ts-2/data/info.pb
I20260929 16:46:48.278954 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 21022
I20260929 16:46:48.279167 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/ts-2/wal/instance
I20260929 16:46:48.279740 789 external_mini_cluster.cc:1428] Setting key 6db66c90521f4268ddd53b380ed37229
I20260929 16:46:48.285442 21151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:48.272112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:32835 (local address 127.0.197.126:46209)
0929 16:46:48.272583 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:48.272590 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:48.273939 (+ 1349us) server_negotiation.cc:407] Connection header received
0929 16:46:48.275911 (+ 1972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:48.275915 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:48.275994 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:48.276143 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:48.277255 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:48.278072 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:48.280763 (+ 2691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:48.281739 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:48.282176 (+ 437us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:48.284151 (+ 1975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:48.284159 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:48.284164 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:46:48.284192 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:48.284255 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:46:48.284269 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:48.284390 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:48.284894 (+ 504us) server_negotiation.cc:299] Negotiation successful
0929 16:46:48.285025 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":215,"threads_started":1}
I20260929 16:46:48.286746 21152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:48.272825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:46209 (local address 127.0.197.67:32835)
0929 16:46:48.273761 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:48.273808 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 16:46:48.275598 (+ 1790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:46:48.276321 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:46:48.276329 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:46:48.276430 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:46:48.277041 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:48.277059 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:48.278355 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:48.278359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:48.280528 (+ 2169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:48.280549 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:48.282109 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:48.282116 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:48.282683 (+ 567us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:48.283722 (+ 1039us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:46:48.283728 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:46:48.283795 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:46:48.283899 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:46:48.284402 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:46:48.284414 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:46:48.284423 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 16:46:48.284865 (+ 442us) client_negotiation.cc:240] Negotiation successful
0929 16:46:48.285354 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":287,"threads_started":1}
I20260929 16:46:48.288563 21149 heartbeater.cc:347] Connected to a master server at 127.0.197.126:46209
I20260929 16:46:48.288920 21149 heartbeater.cc:464] Registering TS with master...
I20260929 16:46:48.289767 21149 heartbeater.cc:511] Master 127.0.197.126:46209 requested a full tablet report, sending...
I20260929 16:46:48.291559 20701 ts_manager.cc:194] Registered new tserver with Master: 3b2bb7d9b01740229bbb67268906df79 (127.0.197.67:37691)
I20260929 16:46:48.292960 20701 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.197.67:32835
I20260929 16:46:48.298303 789 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:46:48.309974 21151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:48.303118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40888 (local address 127.0.197.126:46209)
0929 16:46:48.303322 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:48.303327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:48.303958 (+ 631us) server_negotiation.cc:407] Connection header received
0929 16:46:48.304077 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:48.304082 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:48.304249 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:48.304381 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:48.305057 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:48.305818 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:48.307427 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:48.307875 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:48.307999 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:48.309254 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:48.309259 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:48.309262 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:46:48.309285 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:48.309321 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:46:48.309334 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:48.309422 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:48.309603 (+ 181us) server_negotiation.cc:299] Negotiation successful
0929 16:46:48.309702 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 16:46:48.315475 20701 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40888:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:46:48.322942 20701 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:46:48.348374 21163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:48.337147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52936 (local address 127.0.197.65:37355)
0929 16:46:48.338445 (+ 1298us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:48.338456 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:48.338490 (+ 34us) server_negotiation.cc:407] Connection header received
0929 16:46:48.338641 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:48.338653 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:48.338970 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:48.339986 (+ 1016us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:46:48.340285 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:48.342126 (+ 1841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:48.345412 (+ 3286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:48.346850 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:48.347117 (+ 267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:48.347456 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:48.347759 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 16:46:48.347974 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1098,"thread_start_us":940,"threads_started":1}
I20260929 16:46:48.348709 21162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:48.336885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:37355 (local address 127.0.0.1:52936)
0929 16:46:48.337755 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:48.337795 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 16:46:48.338027 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:46:48.339267 (+ 1240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:46:48.339276 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:46:48.339319 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:46:48.339758 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:48.339772 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:48.342567 (+ 2795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:48.342573 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:48.345112 (+ 2539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:48.345126 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:48.347172 (+ 2046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:48.347178 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:48.347461 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:48.347534 (+ 73us) client_negotiation.cc:769] Sending connection context
0929 16:46:48.347684 (+ 150us) client_negotiation.cc:240] Negotiation successful
0929 16:46:48.347909 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"mutex_wait_us":25,"thread_start_us":358,"threads_started":1}
I20260929 16:46:48.349347 21164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:48.338246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.67:37691 (local address 127.0.0.1:60622)
0929 16:46:48.339000 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:48.339052 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 16:46:48.339179 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:46:48.340477 (+ 1298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:46:48.340484 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:46:48.340514 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:46:48.340919 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:48.340974 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:48.343967 (+ 2993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:48.343976 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:48.346362 (+ 2386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:48.346375 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:48.348302 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:48.348309 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:48.348608 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:48.348690 (+ 82us) client_negotiation.cc:769] Sending connection context
0929 16:46:48.348792 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 16:46:48.348952 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"mutex_wait_us":66,"thread_start_us":502,"threads_started":1}
I20260929 16:46:48.350059 21165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:48.338801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60622 (local address 127.0.197.67:37691)
0929 16:46:48.339802 (+ 1001us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:48.339811 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:48.339846 (+ 35us) server_negotiation.cc:407] Connection header received
0929 16:46:48.339945 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:48.339952 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:48.340233 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:48.341292 (+ 1059us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:46:48.341585 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:48.343788 (+ 2203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:48.346606 (+ 2818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:48.348034 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:48.349102 (+ 1068us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:48.349413 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:48.349493 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 16:46:48.349683 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":782,"thread_start_us":641,"threads_started":1}
I20260929 16:46:48.352669 21166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:48.340185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.66:40849 (local address 127.0.0.1:43548)
0929 16:46:48.342171 (+ 1986us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:48.342197 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 16:46:48.342353 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:46:48.343533 (+ 1180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:46:48.343538 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:46:48.343569 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:46:48.344295 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:48.344308 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:48.346312 (+ 2004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:48.346318 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:48.348695 (+ 2377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:48.348713 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:48.351781 (+ 3068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:48.351788 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:48.351986 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:48.352030 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 16:46:48.352158 (+ 128us) client_negotiation.cc:240] Negotiation successful
0929 16:46:48.352307 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1829,"mutex_wait_us":117,"thread_start_us":1691,"threads_started":1}
I20260929 16:46:48.352779 21167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:48.340851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43548 (local address 127.0.197.66:40849)
0929 16:46:48.342903 (+ 2052us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:48.342910 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:48.342935 (+ 25us) server_negotiation.cc:407] Connection header received
0929 16:46:48.343071 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:48.343081 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:48.343325 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:48.343540 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:46:48.344539 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:48.346130 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:48.350005 (+ 3875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:48.351561 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:48.351737 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:48.351991 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:48.352207 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 16:46:48.352373 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1876,"thread_start_us":1757,"threads_started":1}
I20260929 16:46:48.353484 20822 tablet_service.cc:1467] Processing CreateTablet for tablet 0947c886854d4fb69a4213e12a8adaa4 (DEFAULT_TABLE table=test-table [id=759337c350eb4ac7b2a6c2ac5122501b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:46:48.353963 21084 tablet_service.cc:1467] Processing CreateTablet for tablet 0947c886854d4fb69a4213e12a8adaa4 (DEFAULT_TABLE table=test-table [id=759337c350eb4ac7b2a6c2ac5122501b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:46:48.355399 20822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0947c886854d4fb69a4213e12a8adaa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:46:48.356015 21084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0947c886854d4fb69a4213e12a8adaa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:46:48.356627 20953 tablet_service.cc:1467] Processing CreateTablet for tablet 0947c886854d4fb69a4213e12a8adaa4 (DEFAULT_TABLE table=test-table [id=759337c350eb4ac7b2a6c2ac5122501b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:46:48.357784 20953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0947c886854d4fb69a4213e12a8adaa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:46:48.370499 21168 tablet_bootstrap.cc:492] T 0947c886854d4fb69a4213e12a8adaa4 P 4e811b45a0d74c838cfc5bff0a0fc77b: Bootstrap starting.
I20260929 16:46:48.374209 21168 tablet_bootstrap.cc:654] T 0947c886854d4fb69a4213e12a8adaa4 P 4e811b45a0d74c838cfc5bff0a0fc77b: Neither blocks nor log segments found. Creating new log.
I20260929 16:46:48.375245 21169 tablet_bootstrap.cc:492] T 0947c886854d4fb69a4213e12a8adaa4 P 025e7e81bd6c41b4965c08cf8e94b79e: Bootstrap starting.
I20260929 16:46:48.375735 21168 log.cc:824] T 0947c886854d4fb69a4213e12a8adaa4 P 4e811b45a0d74c838cfc5bff0a0fc77b: Log is configured to *not* fsync() on all Append() calls
I20260929 16:46:48.378798 21170 tablet_bootstrap.cc:492] T 0947c886854d4fb69a4213e12a8adaa4 P 3b2bb7d9b01740229bbb67268906df79: Bootstrap starting.
I20260929 16:46:48.379544 21169 tablet_bootstrap.cc:654] T 0947c886854d4fb69a4213e12a8adaa4 P 025e7e81bd6c41b4965c08cf8e94b79e: Neither blocks nor log segments found. Creating new log.
I20260929 16:46:48.379689 21168 tablet_bootstrap.cc:492] T 0947c886854d4fb69a4213e12a8adaa4 P 4e811b45a0d74c838cfc5bff0a0fc77b: No bootstrap required, opened a new log
I20260929 16:46:48.380018 21168 ts_tablet_manager.cc:1397] T 0947c886854d4fb69a4213e12a8adaa4 P 4e811b45a0d74c838cfc5bff0a0fc77b: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260929 16:46:48.381013 21169 log.cc:824] T 0947c886854d4fb69a4213e12a8adaa4 P 025e7e81bd6c41b4965c08cf8e94b79e: Log is configured to *not* fsync() on all Append() calls
I20260929 16:46:48.382848 21170 tablet_bootstrap.cc:654] T 0947c886854d4fb69a4213e12a8adaa4 P 3b2bb7d9b01740229bbb67268906df79: Neither blocks nor log segments found. Creating new log.
I20260929 16:46:48.384286 21170 log.cc:824] T 0947c886854d4fb69a4213e12a8adaa4 P 3b2bb7d9b01740229bbb67268906df79: Log is configured to *not* fsync() on all Append() calls
I20260929 16:46:48.384900 21169 tablet_bootstrap.cc:492] T 0947c886854d4fb69a4213e12a8adaa4 P 025e7e81bd6c41b4965c08cf8e94b79e: No bootstrap required, opened a new log
I20260929 16:46:48.385329 21169 ts_tablet_manager.cc:1397] T 0947c886854d4fb69a4213e12a8adaa4 P 025e7e81bd6c41b4965c08cf8e94b79e: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.003s
I20260929 16:46:48.388098 21170 tablet_bootstrap.cc:492] T 0947c886854d4fb69a4213e12a8adaa4 P 3b2bb7d9b01740229bbb67268906df79: No bootstrap required, opened a new log
I20260929 16:46:48.388481 21170 ts_tablet_manager.cc:1397] T 0947c886854d4fb69a4213e12a8adaa4 P 3b2bb7d9b01740229bbb67268906df79: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260929 16:46:48.389395 21168 raft_consensus.cc:348] T 0947c886854d4fb69a4213e12a8adaa4 P 4e811b45a0d74c838cfc5bff0a0fc77b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e7e81bd6c41b4965c08cf8e94b79e" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 37355 } } peers { permanent_uuid: "3b2bb7d9b01740229bbb67268906df79" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 37691 } } peers { permanent_uuid: "4e811b45a0d74c838cfc5bff0a0fc77b" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 40849 } }
I20260929 16:46:48.389958 21168 raft_consensus.cc:374] T 0947c886854d4fb69a4213e12a8adaa4 P 4e811b45a0d74c838cfc5bff0a0fc77b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:46:48.390110 21168 raft_consensus.cc:729] T 0947c886854d4fb69a4213e12a8adaa4 P 4e811b45a0d74c838cfc5bff0a0fc77b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e811b45a0d74c838cfc5bff0a0fc77b, State: Initialized, Role: FOLLOWER
I20260929 16:46:48.390818 21168 consensus_queue.cc:260] T 0947c886854d4fb69a4213e12a8adaa4 P 4e811b45a0d74c838cfc5bff0a0fc77b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e7e81bd6c41b4965c08cf8e94b79e" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 37355 } } peers { permanent_uuid: "3b2bb7d9b01740229bbb67268906df79" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 37691 } } peers { permanent_uuid: "4e811b45a0d74c838cfc5bff0a0fc77b" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 40849 } }
I20260929 16:46:48.392170 21018 heartbeater.cc:503] Master 127.0.197.126:46209 was elected leader, sending a full tablet report...
I20260929 16:46:48.392756 21168 ts_tablet_manager.cc:1428] T 0947c886854d4fb69a4213e12a8adaa4 P 4e811b45a0d74c838cfc5bff0a0fc77b: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260929 16:46:48.396801 21169 raft_consensus.cc:348] T 0947c886854d4fb69a4213e12a8adaa4 P 025e7e81bd6c41b4965c08cf8e94b79e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e7e81bd6c41b4965c08cf8e94b79e" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 37355 } } peers { permanent_uuid: "3b2bb7d9b01740229bbb67268906df79" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 37691 } } peers { permanent_uuid: "4e811b45a0d74c838cfc5bff0a0fc77b" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 40849 } }
I20260929 16:46:48.397377 21169 raft_consensus.cc:374] T 0947c886854d4fb69a4213e12a8adaa4 P 025e7e81bd6c41b4965c08cf8e94b79e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:46:48.397487 21169 raft_consensus.cc:729] T 0947c886854d4fb69a4213e12a8adaa4 P 025e7e81bd6c41b4965c08cf8e94b79e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 025e7e81bd6c41b4965c08cf8e94b79e, State: Initialized, Role: FOLLOWER
I20260929 16:46:48.398222 21169 consensus_queue.cc:260] T 0947c886854d4fb69a4213e12a8adaa4 P 025e7e81bd6c41b4965c08cf8e94b79e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e7e81bd6c41b4965c08cf8e94b79e" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 37355 } } peers { permanent_uuid: "3b2bb7d9b01740229bbb67268906df79" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 37691 } } peers { permanent_uuid: "4e811b45a0d74c838cfc5bff0a0fc77b" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 40849 } }
I20260929 16:46:48.400913 21169 ts_tablet_manager.cc:1428] T 0947c886854d4fb69a4213e12a8adaa4 P 025e7e81bd6c41b4965c08cf8e94b79e: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
I20260929 16:46:48.402587 21170 raft_consensus.cc:348] T 0947c886854d4fb69a4213e12a8adaa4 P 3b2bb7d9b01740229bbb67268906df79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e7e81bd6c41b4965c08cf8e94b79e" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 37355 } } peers { permanent_uuid: "3b2bb7d9b01740229bbb67268906df79" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 37691 } } peers { permanent_uuid: "4e811b45a0d74c838cfc5bff0a0fc77b" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 40849 } }
I20260929 16:46:48.403771 21170 raft_consensus.cc:374] T 0947c886854d4fb69a4213e12a8adaa4 P 3b2bb7d9b01740229bbb67268906df79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:46:48.403934 21170 raft_consensus.cc:729] T 0947c886854d4fb69a4213e12a8adaa4 P 3b2bb7d9b01740229bbb67268906df79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b2bb7d9b01740229bbb67268906df79, State: Initialized, Role: FOLLOWER
I20260929 16:46:48.404767 21170 consensus_queue.cc:260] T 0947c886854d4fb69a4213e12a8adaa4 P 3b2bb7d9b01740229bbb67268906df79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e7e81bd6c41b4965c08cf8e94b79e" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 37355 } } peers { permanent_uuid: "3b2bb7d9b01740229bbb67268906df79" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 37691 } } peers { permanent_uuid: "4e811b45a0d74c838cfc5bff0a0fc77b" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 40849 } }
I20260929 16:46:48.406013 21149 heartbeater.cc:503] Master 127.0.197.126:46209 was elected leader, sending a full tablet report...
I20260929 16:46:48.406704 21170 ts_tablet_manager.cc:1428] T 0947c886854d4fb69a4213e12a8adaa4 P 3b2bb7d9b01740229bbb67268906df79: Time spent starting tablet: real 0.018s user 0.011s sys 0.004s
I20260929 16:46:48.436376 21176 raft_consensus.cc:482] T 0947c886854d4fb69a4213e12a8adaa4 P 3b2bb7d9b01740229bbb67268906df79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:46:48.436784 21176 raft_consensus.cc:504] T 0947c886854d4fb69a4213e12a8adaa4 P 3b2bb7d9b01740229bbb67268906df79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e7e81bd6c41b4965c08cf8e94b79e" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 37355 } } peers { permanent_uuid: "3b2bb7d9b01740229bbb67268906df79" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 37691 } } peers { permanent_uuid: "4e811b45a0d74c838cfc5bff0a0fc77b" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 40849 } }
I20260929 16:46:48.438663 21176 leader_election.cc:288] T 0947c886854d4fb69a4213e12a8adaa4 P 3b2bb7d9b01740229bbb67268906df79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 025e7e81bd6c41b4965c08cf8e94b79e (127.0.197.65:37355), 4e811b45a0d74c838cfc5bff0a0fc77b (127.0.197.66:40849)
I20260929 16:46:48.446564 21163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:48.439328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:35353 (local address 127.0.197.65:37355)
0929 16:46:48.439549 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:48.439556 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:48.439577 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:46:48.439662 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:48.439667 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:48.439768 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:48.439889 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:46:48.440731 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:48.441784 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:48.444329 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:48.445588 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:48.445755 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:48.445812 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:48.446256 (+ 444us) server_negotiation.cc:299] Negotiation successful
0929 16:46:48.446368 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 16:46:48.446683 21152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:48.439080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.65:37355 (local address 127.0.197.67:35353)
0929 16:46:48.439335 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:48.439359 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 16:46:48.439491 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:46:48.440115 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:46:48.440123 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:46:48.440165 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:46:48.440497 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:48.440510 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:48.442008 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:48.442014 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:48.444132 (+ 2118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:48.444150 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:48.445773 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:48.445780 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:48.446003 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:48.446049 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 16:46:48.446164 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 16:46:48.446341 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260929 16:46:48.448063 21167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:48.440030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.197.67:48379 (local address 127.0.197.66:40849)
0929 16:46:48.440245 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:48.440251 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:48.440945 (+ 694us) server_negotiation.cc:407] Connection header received
0929 16:46:48.441190 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:48.441195 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:48.441278 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:48.441371 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:46:48.442053 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:48.443259 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:48.445753 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:48.447179 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:48.447389 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:48.447466 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:48.447715 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 16:46:48.447822 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 16:46:48.448143 21177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:48.439530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.66:40849 (local address 127.0.197.67:48379)
0929 16:46:48.440782 (+ 1252us) negotiation.cc:107] Waiting for socket to connect
0929 16:46:48.440813 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 16:46:48.440955 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:46:48.441424 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:46:48.441429 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:46:48.441458 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:46:48.441837 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:48.441851 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:48.443444 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:48.443448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:48.445451 (+ 2003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:46:48.445471 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:46:48.447374 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:46:48.447378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:46:48.447504 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:48.447529 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 16:46:48.447681 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 16:46:48.447831 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1129,"thread_start_us":910,"threads_started":1}
I20260929 16:46:48.448199 20842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0947c886854d4fb69a4213e12a8adaa4" candidate_uuid: "3b2bb7d9b01740229bbb67268906df79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "025e7e81bd6c41b4965c08cf8e94b79e" is_pre_election: true
I20260929 16:46:48.448899 20842 raft_consensus.cc:2454] T 0947c886854d4fb69a4213e12a8adaa4 P 025e7e81bd6c41b4965c08cf8e94b79e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b2bb7d9b01740229bbb67268906df79 in term 0.
I20260929 16:46:48.449486 20973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0947c886854d4fb69a4213e12a8adaa4" candidate_uuid: "3b2bb7d9b01740229bbb67268906df79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e811b45a0d74c838cfc5bff0a0fc77b" is_pre_election: true
I20260929 16:46:48.449685 21037 leader_election.cc:302] T 0947c886854d4fb69a4213e12a8adaa4 P 3b2bb7d9b01740229bbb67268906df79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 025e7e81bd6c41b4965c08cf8e94b79e, 3b2bb7d9b01740229bbb67268906df79; no voters:
I20260929 16:46:48.450130 21176 raft_consensus.cc:2781] T 0947c886854d4fb69a4213e12a8adaa4 P 3b2bb7d9b01740229bbb67268906df79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:46:48.450088 20973 raft_consensus.cc:2454] T 0947c886854d4fb69a4213e12a8adaa4 P 4e811b45a0d74c838cfc5bff0a0fc77b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b2bb7d9b01740229bbb67268906df79 in term 0.
I20260929 16:46:48.450309 21176 raft_consensus.cc:482] T 0947c886854d4fb69a4213e12a8adaa4 P 3b2bb7d9b01740229bbb67268906df79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:46:48.450429 21176 raft_consensus.cc:3037] T 0947c886854d4fb69a4213e12a8adaa4 P 3b2bb7d9b01740229bbb67268906df79 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:46:48.452997 21176 raft_consensus.cc:504] T 0947c886854d4fb69a4213e12a8adaa4 P 3b2bb7d9b01740229bbb67268906df79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e7e81bd6c41b4965c08cf8e94b79e" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 37355 } } peers { permanent_uuid: "3b2bb7d9b01740229bbb67268906df79" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 37691 } } peers { permanent_uuid: "4e811b45a0d74c838cfc5bff0a0fc77b" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 40849 } }
I20260929 16:46:48.454180 21176 leader_election.cc:288] T 0947c886854d4fb69a4213e12a8adaa4 P 3b2bb7d9b01740229bbb67268906df79 [CANDIDATE]: Term 1 election: Requested vote from peers 025e7e81bd6c41b4965c08cf8e94b79e (127.0.197.65:37355), 4e811b45a0d74c838cfc5bff0a0fc77b (127.0.197.66:40849)
I20260929 16:46:48.455111 20842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0947c886854d4fb69a4213e12a8adaa4" candidate_uuid: "3b2bb7d9b01740229bbb67268906df79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "025e7e81bd6c41b4965c08cf8e94b79e"
I20260929 16:46:48.455103 20973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0947c886854d4fb69a4213e12a8adaa4" candidate_uuid: "3b2bb7d9b01740229bbb67268906df79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e811b45a0d74c838cfc5bff0a0fc77b"
I20260929 16:46:48.455456 20842 raft_consensus.cc:3037] T 0947c886854d4fb69a4213e12a8adaa4 P 025e7e81bd6c41b4965c08cf8e94b79e [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:46:48.455615 20973 raft_consensus.cc:3037] T 0947c886854d4fb69a4213e12a8adaa4 P 4e811b45a0d74c838cfc5bff0a0fc77b [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:46:48.458117 20973 raft_consensus.cc:2454] T 0947c886854d4fb69a4213e12a8adaa4 P 4e811b45a0d74c838cfc5bff0a0fc77b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b2bb7d9b01740229bbb67268906df79 in term 1.
I20260929 16:46:48.458132 20842 raft_consensus.cc:2454] T 0947c886854d4fb69a4213e12a8adaa4 P 025e7e81bd6c41b4965c08cf8e94b79e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b2bb7d9b01740229bbb67268906df79 in term 1.
I20260929 16:46:48.458712 21037 leader_election.cc:302] T 0947c886854d4fb69a4213e12a8adaa4 P 3b2bb7d9b01740229bbb67268906df79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 025e7e81bd6c41b4965c08cf8e94b79e, 3b2bb7d9b01740229bbb67268906df79; no voters:
I20260929 16:46:48.459187 21176 raft_consensus.cc:2781] T 0947c886854d4fb69a4213e12a8adaa4 P 3b2bb7d9b01740229bbb67268906df79 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:46:48.461421 21176 raft_consensus.cc:686] T 0947c886854d4fb69a4213e12a8adaa4 P 3b2bb7d9b01740229bbb67268906df79 [term 1 LEADER]: Becoming Leader. State: Replica: 3b2bb7d9b01740229bbb67268906df79, State: Running, Role: LEADER
I20260929 16:46:48.462388 21176 consensus_queue.cc:237] T 0947c886854d4fb69a4213e12a8adaa4 P 3b2bb7d9b01740229bbb67268906df79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e7e81bd6c41b4965c08cf8e94b79e" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 37355 } } peers { permanent_uuid: "3b2bb7d9b01740229bbb67268906df79" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 37691 } } peers { permanent_uuid: "4e811b45a0d74c838cfc5bff0a0fc77b" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 40849 } }
I20260929 16:46:48.467765 20700 catalog_manager.cc:5495] T 0947c886854d4fb69a4213e12a8adaa4 P 3b2bb7d9b01740229bbb67268906df79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b2bb7d9b01740229bbb67268906df79 (127.0.197.67). New cstate: current_term: 1 leader_uuid: "3b2bb7d9b01740229bbb67268906df79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "025e7e81bd6c41b4965c08cf8e94b79e" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 37355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b2bb7d9b01740229bbb67268906df79" member_type: VOTER last_known_addr { host: "127.0.197.67" port: 37691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e811b45a0d74c838cfc5bff0a0fc77b" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 40849 } health_report { overall_health: UNKNOWN } } }
W20260929 16:46:48.469007 20888 tablet.cc:2369] T 0947c886854d4fb69a4213e12a8adaa4 P 025e7e81bd6c41b4965c08cf8e94b79e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:46:48.499302 21019 tablet.cc:2369] T 0947c886854d4fb69a4213e12a8adaa4 P 4e811b45a0d74c838cfc5bff0a0fc77b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:46:48.509466 21165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:48.503113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60630 (local address 127.0.197.67:37691)
0929 16:46:48.503453 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:48.503457 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:48.503475 (+ 18us) server_negotiation.cc:407] Connection header received
0929 16:46:48.503614 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:48.503618 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:48.503704 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:48.503833 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:46:48.504276 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:48.505030 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:48.506507 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:48.506954 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:48.507162 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:48.508035 (+ 873us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:46:48.508625 (+ 590us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:46:48.508765 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:48.509038 (+ 273us) server_negotiation.cc:299] Negotiation successful
0929 16:46:48.509116 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222}
I20260929 16:46:48.519876 20973 raft_consensus.cc:1261] T 0947c886854d4fb69a4213e12a8adaa4 P 4e811b45a0d74c838cfc5bff0a0fc77b [term 1 FOLLOWER]: Refusing update from remote peer 3b2bb7d9b01740229bbb67268906df79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:46:48.519876 20842 raft_consensus.cc:1261] T 0947c886854d4fb69a4213e12a8adaa4 P 025e7e81bd6c41b4965c08cf8e94b79e [term 1 FOLLOWER]: Refusing update from remote peer 3b2bb7d9b01740229bbb67268906df79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260929 16:46:48.521070 21150 tablet.cc:2369] T 0947c886854d4fb69a4213e12a8adaa4 P 3b2bb7d9b01740229bbb67268906df79: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:46:48.521471 21178 consensus_queue.cc:1035] T 0947c886854d4fb69a4213e12a8adaa4 P 3b2bb7d9b01740229bbb67268906df79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "025e7e81bd6c41b4965c08cf8e94b79e" member_type: VOTER last_known_addr { host: "127.0.197.65" port: 37355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:46:48.522243 21176 consensus_queue.cc:1035] T 0947c886854d4fb69a4213e12a8adaa4 P 3b2bb7d9b01740229bbb67268906df79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e811b45a0d74c838cfc5bff0a0fc77b" member_type: VOTER last_known_addr { host: "127.0.197.66" port: 40849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:46:48.536628 21183 mvcc.cc:205] Tried to move back new op lower bound from 7334708873284243456 to 7334708873067683840. Current Snapshot: MvccSnapshot[applied={T|T < 7334708873284243456}]
I20260929 16:46:48.555104 20701 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:46:48.555488 20701 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:46:48.559819 20701 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f66f7d6a8d194b1c895598c8f2cef207: Sending DeleteTablet for 3 replicas of tablet 0947c886854d4fb69a4213e12a8adaa4
I20260929 16:46:48.561657 21084 tablet_service.cc:1514] Processing DeleteTablet for tablet 0947c886854d4fb69a4213e12a8adaa4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:46:48 UTC) from {username='slave'} at 127.0.0.1:60622
I20260929 16:46:48.561643 20953 tablet_service.cc:1514] Processing DeleteTablet for tablet 0947c886854d4fb69a4213e12a8adaa4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:46:48 UTC) from {username='slave'} at 127.0.0.1:43548
I20260929 16:46:48.562987 20822 tablet_service.cc:1514] Processing DeleteTablet for tablet 0947c886854d4fb69a4213e12a8adaa4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:46:48 UTC) from {username='slave'} at 127.0.0.1:52936
I20260929 16:46:48.563982 21193 tablet_replica.cc:330] T 0947c886854d4fb69a4213e12a8adaa4 P 025e7e81bd6c41b4965c08cf8e94b79e: stopping tablet replica
I20260929 16:46:48.564648 21193 raft_consensus.cc:2229] T 0947c886854d4fb69a4213e12a8adaa4 P 025e7e81bd6c41b4965c08cf8e94b79e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:46:48.565027 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 20760
I20260929 16:46:48.565459 21193 raft_consensus.cc:2258] T 0947c886854d4fb69a4213e12a8adaa4 P 025e7e81bd6c41b4965c08cf8e94b79e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:46:48.570030 21193 ts_tablet_manager.cc:1905] T 0947c886854d4fb69a4213e12a8adaa4 P 025e7e81bd6c41b4965c08cf8e94b79e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:46:48.575984 21192 tablet_replica.cc:330] T 0947c886854d4fb69a4213e12a8adaa4 P 3b2bb7d9b01740229bbb67268906df79: stopping tablet replica
I20260929 16:46:48.576650 21192 raft_consensus.cc:2229] T 0947c886854d4fb69a4213e12a8adaa4 P 3b2bb7d9b01740229bbb67268906df79 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:46:48.577543 21192 raft_consensus.cc:2258] T 0947c886854d4fb69a4213e12a8adaa4 P 3b2bb7d9b01740229bbb67268906df79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:46:48.578943 21193 ts_tablet_manager.cc:1918] T 0947c886854d4fb69a4213e12a8adaa4 P 025e7e81bd6c41b4965c08cf8e94b79e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:46:48.579231 21193 log.cc:1196] T 0947c886854d4fb69a4213e12a8adaa4 P 025e7e81bd6c41b4965c08cf8e94b79e: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/ts-0/wal/wals/0947c886854d4fb69a4213e12a8adaa4
I20260929 16:46:48.579892 21193 ts_tablet_manager.cc:1939] T 0947c886854d4fb69a4213e12a8adaa4 P 025e7e81bd6c41b4965c08cf8e94b79e: Deleting consensus metadata
I20260929 16:46:48.580904 21191 tablet_replica.cc:330] T 0947c886854d4fb69a4213e12a8adaa4 P 4e811b45a0d74c838cfc5bff0a0fc77b: stopping tablet replica
I20260929 16:46:48.581368 21191 raft_consensus.cc:2229] T 0947c886854d4fb69a4213e12a8adaa4 P 4e811b45a0d74c838cfc5bff0a0fc77b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:46:48.582278 20685 catalog_manager.cc:4841] TS 025e7e81bd6c41b4965c08cf8e94b79e (127.0.197.65:37355): tablet 0947c886854d4fb69a4213e12a8adaa4 (table test-table [id=759337c350eb4ac7b2a6c2ac5122501b]) successfully deleted
I20260929 16:46:48.582581 21192 ts_tablet_manager.cc:1905] T 0947c886854d4fb69a4213e12a8adaa4 P 3b2bb7d9b01740229bbb67268906df79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:46:48.582700 21191 raft_consensus.cc:2258] T 0947c886854d4fb69a4213e12a8adaa4 P 4e811b45a0d74c838cfc5bff0a0fc77b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:46:48.584831 21191 ts_tablet_manager.cc:1905] T 0947c886854d4fb69a4213e12a8adaa4 P 4e811b45a0d74c838cfc5bff0a0fc77b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:46:48.587937 21163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:48.566452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52944 (local address 127.0.197.65:37355)
0929 16:46:48.566655 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:48.566662 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:48.572345 (+ 5683us) server_negotiation.cc:407] Connection header received
0929 16:46:48.573368 (+ 1023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:48.573374 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:48.573484 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:48.573562 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:48.574623 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:48.575776 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:48.577795 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:48.578466 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:48.578595 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:48.582764 (+ 4169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:48.582776 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:48.582795 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:46:48.584360 (+ 1565us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:48.584435 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:46:48.584465 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:48.584596 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:48.587371 (+ 2775us) server_negotiation.cc:299] Negotiation successful
0929 16:46:48.587531 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 16:46:48.588815 20882 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:46:48.591418 21192 ts_tablet_manager.cc:1918] T 0947c886854d4fb69a4213e12a8adaa4 P 3b2bb7d9b01740229bbb67268906df79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:46:48.591753 21192 log.cc:1196] T 0947c886854d4fb69a4213e12a8adaa4 P 3b2bb7d9b01740229bbb67268906df79: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/ts-2/wal/wals/0947c886854d4fb69a4213e12a8adaa4
I20260929 16:46:48.592430 21192 ts_tablet_manager.cc:1939] T 0947c886854d4fb69a4213e12a8adaa4 P 3b2bb7d9b01740229bbb67268906df79: Deleting consensus metadata
I20260929 16:46:48.592881 21191 ts_tablet_manager.cc:1918] T 0947c886854d4fb69a4213e12a8adaa4 P 4e811b45a0d74c838cfc5bff0a0fc77b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:46:48.593073 21191 log.cc:1196] T 0947c886854d4fb69a4213e12a8adaa4 P 4e811b45a0d74c838cfc5bff0a0fc77b: Deleting WAL directory at /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790700068255611-789-0/minicluster-data/ts-1/wal/wals/0947c886854d4fb69a4213e12a8adaa4
I20260929 16:46:48.593559 21191 ts_tablet_manager.cc:1939] T 0947c886854d4fb69a4213e12a8adaa4 P 4e811b45a0d74c838cfc5bff0a0fc77b: Deleting consensus metadata
I20260929 16:46:48.594880 20686 catalog_manager.cc:4841] TS 3b2bb7d9b01740229bbb67268906df79 (127.0.197.67:37691): tablet 0947c886854d4fb69a4213e12a8adaa4 (table test-table [id=759337c350eb4ac7b2a6c2ac5122501b]) successfully deleted
I20260929 16:46:48.595356 20685 catalog_manager.cc:4841] TS 4e811b45a0d74c838cfc5bff0a0fc77b (127.0.197.66:40849): tablet 0947c886854d4fb69a4213e12a8adaa4 (table test-table [id=759337c350eb4ac7b2a6c2ac5122501b]) successfully deleted
I20260929 16:46:48.692644 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 20760
I20260929 16:46:48.718895 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 20891
I20260929 16:46:48.727684 21167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:48.719819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43556 (local address 127.0.197.66:40849)
0929 16:46:48.720000 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:48.720005 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:48.720024 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:46:48.720233 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:48.720237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:48.720346 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:48.720455 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:48.720968 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:48.721744 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:48.723891 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:48.724337 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:48.724449 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:48.726130 (+ 1681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:48.726139 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:48.726155 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:46:48.726777 (+ 622us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:48.726835 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:46:48.726858 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:48.726958 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:48.727166 (+ 208us) server_negotiation.cc:299] Negotiation successful
0929 16:46:48.727277 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:46:48.728523 21013 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:46:48.830395 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 20891
I20260929 16:46:48.851399 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 21022
I20260929 16:46:48.860925 21165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:48.852373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60632 (local address 127.0.197.67:37691)
0929 16:46:48.852583 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:48.852590 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:48.852612 (+ 22us) server_negotiation.cc:407] Connection header received
0929 16:46:48.852684 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:48.852689 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:48.852794 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:48.852925 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:48.853693 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:48.855082 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:48.857266 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:48.857752 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:48.857882 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:48.859354 (+ 1472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:48.859365 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:48.859380 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:46:48.860034 (+ 654us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:48.860097 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:46:48.860120 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:48.860214 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:48.860409 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 16:46:48.860511 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 16:46:48.861860 21144 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:46:48.987445 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 21022
I20260929 16:46:49.009563 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 20670
I20260929 16:46:49.010850 20731 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:46:49.126420 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 20670
2026-09-29T16:46:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3088 ms)
[----------] 6 tests from EncryptionPolicyTest (23695 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790700068255611-789-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:46:49 dist-test-slave-l31f krb5kdc[21201](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:46:49 dist-test-slave-l31f krb5kdc[21201](info): set up 2 sockets
Sep 29 16:46:49 dist-test-slave-l31f krb5kdc[21201](info): commencing operation
krb5kdc: starting...
W20260929 16:46:50.291729 789 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.005s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790700068255611-789-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790700068255611-789-0/krb5kdc/test-user.keytab.
Sep 29 16:46:50 dist-test-slave-l31f krb5kdc[21201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:46:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:46:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790700068255611-789-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790700068255611-789-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:46:50.571060 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44827
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790700068255611-789-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:46:50.927138 21214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:46:50.927467 21214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:46:50.927562 21214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:46:50.935139 21214 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:46:50.935285 21214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:46:50.935341 21214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:46:50.935407 21214 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:46:50.935451 21214 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:46:50.945914 21214 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790700068255611-789-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44827
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790700068255611-789-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:46:50.947283 21214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:46:50.948601 21214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:46:50.961660 21220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:50.961660 21219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:50.961694 21222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:46:50.962677 21214 server_base.cc:1034] running on GCE node
I20260929 16:46:50.963694 21214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:46:50.965399 21214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:46:50.966675 21214 hybrid_clock.cc:648] HybridClock initialized: now 1790700410966627 us; error 68 us; skew 500 ppm
I20260929 16:46:50.967193 21214 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:46:50 dist-test-slave-l31f krb5kdc[21201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:46:50.970690 21214 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:46:50.972571 21214 webserver.cc:460] Webserver started at http://127.0.0.1:34731/ using document root <none> and password file <none>
I20260929 16:46:50.973418 21214 fs_manager.cc:359] Metadata directory not provided
I20260929 16:46:50.973569 21214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:46:50.973929 21214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:46:50.977272 21214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790700068255611-789-0/minicluster-data/master-0/data/instance:
uuid: "31be286d7bbb402c9da3d3a7786cc233"
format_stamp: "Formatted at 2026-09-29 16:46:50 on dist-test-slave-l31f"
server_key: "e26a9555339e2e1aca859aef99f48365"
server_key_iv: "82e287de316c13fbfb190ea22acab077"
server_key_version: "encryptionkey@0"
I20260929 16:46:50.978149 21214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790700068255611-789-0/minicluster-data/master-0/wal/instance:
uuid: "31be286d7bbb402c9da3d3a7786cc233"
format_stamp: "Formatted at 2026-09-29 16:46:50 on dist-test-slave-l31f"
server_key: "e26a9555339e2e1aca859aef99f48365"
server_key_iv: "82e287de316c13fbfb190ea22acab077"
server_key_version: "encryptionkey@0"
I20260929 16:46:50.984109 21214 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 16:46:50.988076 21229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:50.989832 21214 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260929 16:46:50.990078 21214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "31be286d7bbb402c9da3d3a7786cc233"
format_stamp: "Formatted at 2026-09-29 16:46:50 on dist-test-slave-l31f"
server_key: "e26a9555339e2e1aca859aef99f48365"
server_key_iv: "82e287de316c13fbfb190ea22acab077"
server_key_version: "encryptionkey@0"
I20260929 16:46:50.990337 21214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:46:51.013535 21214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:46:51.018661 21214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:46:51.019094 21214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:46:51.039932 21214 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44827
I20260929 16:46:51.039985 21280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44827 every 8 connection(s)
I20260929 16:46:51.041522 21214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:46:51.045626 21281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:46:51.048996 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 21214
I20260929 16:46:51.049261 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790700068255611-789-0/minicluster-data/master-0/wal/instance
I20260929 16:46:51.049933 789 external_mini_cluster.cc:1428] Setting key c840bf7f19b40430e0afb0c5b3dea94f
I20260929 16:46:51.059607 21281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31be286d7bbb402c9da3d3a7786cc233: Bootstrap starting.
I20260929 16:46:51.063872 21281 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31be286d7bbb402c9da3d3a7786cc233: Neither blocks nor log segments found. Creating new log.
I20260929 16:46:51.065414 21281 log.cc:824] T 00000000000000000000000000000000 P 31be286d7bbb402c9da3d3a7786cc233: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:46:51 dist-test-slave-l31f krb5kdc[21201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 16:46:51.068920 21281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31be286d7bbb402c9da3d3a7786cc233: No bootstrap required, opened a new log
I20260929 16:46:51.076244 21284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:51.052195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43181 (local address 127.0.0.1:44827)
0929 16:46:51.053055 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:51.053067 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:51.053109 (+ 42us) server_negotiation.cc:407] Connection header received
0929 16:46:51.054843 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:51.054869 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:51.055414 (+ 545us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:51.055786 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:51.057692 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:51.059222 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:51.061034 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:51.061772 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:51.061925 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:51.067435 (+ 5510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:51.067445 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:51.067463 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:46:51.067515 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:51.070959 (+ 3444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:51.072288 (+ 1329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:51.072295 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:51.072306 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:51.072521 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:51.073075 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:51.073079 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:51.073081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:51.073746 (+ 665us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:51.074008 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:51.074571 (+ 563us) server_negotiation.cc:299] Negotiation successful
0929 16:46:51.074995 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":400,"threads_started":1}
I20260929 16:46:51.078706 21281 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31be286d7bbb402c9da3d3a7786cc233 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31be286d7bbb402c9da3d3a7786cc233" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44827 } }
I20260929 16:46:51.079219 21281 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31be286d7bbb402c9da3d3a7786cc233 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:46:51.079341 21281 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31be286d7bbb402c9da3d3a7786cc233 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31be286d7bbb402c9da3d3a7786cc233, State: Initialized, Role: FOLLOWER
I20260929 16:46:51.080068 21281 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31be286d7bbb402c9da3d3a7786cc233 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31be286d7bbb402c9da3d3a7786cc233" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44827 } }
I20260929 16:46:51.080353 21281 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31be286d7bbb402c9da3d3a7786cc233 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:46:51.080507 21281 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31be286d7bbb402c9da3d3a7786cc233 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:46:51.080708 21281 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31be286d7bbb402c9da3d3a7786cc233 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:46:51.080905 789 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:46:51.083426 21281 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31be286d7bbb402c9da3d3a7786cc233 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31be286d7bbb402c9da3d3a7786cc233" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44827 } }
I20260929 16:46:51.084180 21281 leader_election.cc:302] T 00000000000000000000000000000000 P 31be286d7bbb402c9da3d3a7786cc233 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31be286d7bbb402c9da3d3a7786cc233; no voters:
I20260929 16:46:51.084956 21281 leader_election.cc:288] T 00000000000000000000000000000000 P 31be286d7bbb402c9da3d3a7786cc233 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:46:51.085086 21290 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31be286d7bbb402c9da3d3a7786cc233 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:46:51.087953 21281 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31be286d7bbb402c9da3d3a7786cc233 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:46:51.091329 21290 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31be286d7bbb402c9da3d3a7786cc233 [term 1 LEADER]: Becoming Leader. State: Replica: 31be286d7bbb402c9da3d3a7786cc233, State: Running, Role: LEADER
I20260929 16:46:51.092271 21290 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31be286d7bbb402c9da3d3a7786cc233 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31be286d7bbb402c9da3d3a7786cc233" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44827 } }
I20260929 16:46:51.096357 21292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31be286d7bbb402c9da3d3a7786cc233 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31be286d7bbb402c9da3d3a7786cc233. Latest consensus state: current_term: 1 leader_uuid: "31be286d7bbb402c9da3d3a7786cc233" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31be286d7bbb402c9da3d3a7786cc233" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44827 } } }
I20260929 16:46:51.096911 21292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31be286d7bbb402c9da3d3a7786cc233 [sys.catalog]: This master's current role is: LEADER
I20260929 16:46:51.097710 21291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31be286d7bbb402c9da3d3a7786cc233 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31be286d7bbb402c9da3d3a7786cc233" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31be286d7bbb402c9da3d3a7786cc233" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44827 } } }
I20260929 16:46:51.098031 21291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31be286d7bbb402c9da3d3a7786cc233 [sys.catalog]: This master's current role is: LEADER
I20260929 16:46:51.108533 21284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:51.086390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59335 (local address 127.0.0.1:44827)
0929 16:46:51.086639 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:51.086649 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:51.087248 (+ 599us) server_negotiation.cc:407] Connection header received
0929 16:46:51.087529 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:51.087535 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:51.087643 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:51.087717 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:51.089429 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:51.090803 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:51.092443 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:51.093240 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:51.093460 (+ 220us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:51.097452 (+ 3992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:51.097462 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:51.097477 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:46:51.097535 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:51.102592 (+ 5057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:51.104175 (+ 1583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:51.104184 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:51.104190 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:51.104405 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:51.105151 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:51.105159 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:51.105165 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:51.105772 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:51.106125 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:51.107711 (+ 1586us) server_negotiation.cc:299] Negotiation successful
0929 16:46:51.108005 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 16:46:51.108796 21300 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:46:51.115397 21300 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:46:51.125893 21300 catalog_manager.cc:1347] Generated new cluster ID: acf291d174f148368f6bf0465a13c4da
I20260929 16:46:51.126036 21300 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:46:51.148401 21300 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:46:51.150377 21300 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:46:51.167080 21300 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31be286d7bbb402c9da3d3a7786cc233: Generated new TSK 0
I20260929 16:46:51.168026 21300 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:46:51.175740 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 21214
I20260929 16:46:51.177552 21276 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:46:51.276058 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 21214
W20260929 16:46:51.294744 21199 connection.cc:535] client connection to 127.0.0.1:44827 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44827): Connection reset by peer (error 104)
2026-09-29T16:46:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2143 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790700068255611-789-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:46:51 dist-test-slave-l31f krb5kdc[21314](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:46:51 dist-test-slave-l31f krb5kdc[21314](info): set up 2 sockets
Sep 29 16:46:51 dist-test-slave-l31f krb5kdc[21314](info): commencing operation
krb5kdc: starting...
W20260929 16:46:52.407564 789 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.007s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790700068255611-789-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790700068255611-789-0/krb5kdc/test-user.keytab.
Sep 29 16:46:52 dist-test-slave-l31f krb5kdc[21314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:46:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:46:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790700068255611-789-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790700068255611-789-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:46:52.686966 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40603
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790700068255611-789-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40603
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:46:53.055495 21327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:46:53.055884 21327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:46:53.056071 21327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:46:53.064527 21327 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:46:53.064714 21327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:46:53.064826 21327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:46:53.064924 21327 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:46:53.065023 21327 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:46:53.075531 21327 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790700068255611-789-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40603
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790700068255611-789-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:46:53.076984 21327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:46:53.078356 21327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:46:53.091365 21332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:53.091452 21335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:53.091388 21333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:46:53.092592 21327 server_base.cc:1034] running on GCE node
I20260929 16:46:53.093489 21327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:46:53.095211 21327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:46:53.096493 21327 hybrid_clock.cc:648] HybridClock initialized: now 1790700413096450 us; error 48 us; skew 500 ppm
I20260929 16:46:53.096952 21327 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:46:53 dist-test-slave-l31f krb5kdc[21314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700413, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:46:53.105571 21327 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:46:53.107527 21327 webserver.cc:460] Webserver started at http://127.0.0.1:46183/ using document root <none> and password file <none>
I20260929 16:46:53.108325 21327 fs_manager.cc:359] Metadata directory not provided
I20260929 16:46:53.108502 21327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:46:53.108841 21327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:46:53.112207 21327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790700068255611-789-0/minicluster-data/master-0/data/instance:
uuid: "8c911fd713d14715b77e2793aac3e3a4"
format_stamp: "Formatted at 2026-09-29 16:46:53 on dist-test-slave-l31f"
server_key: "d4249a9202559cc806c977be085d7cf3"
server_key_iv: "93e6dad91872989ff42bf05f302ac156"
server_key_version: "encryptionkey@0"
I20260929 16:46:53.113250 21327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790700068255611-789-0/minicluster-data/master-0/wal/instance:
uuid: "8c911fd713d14715b77e2793aac3e3a4"
format_stamp: "Formatted at 2026-09-29 16:46:53 on dist-test-slave-l31f"
server_key: "d4249a9202559cc806c977be085d7cf3"
server_key_iv: "93e6dad91872989ff42bf05f302ac156"
server_key_version: "encryptionkey@0"
I20260929 16:46:53.119436 21327 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260929 16:46:53.123805 21342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:53.125597 21327 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:46:53.125873 21327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "8c911fd713d14715b77e2793aac3e3a4"
format_stamp: "Formatted at 2026-09-29 16:46:53 on dist-test-slave-l31f"
server_key: "d4249a9202559cc806c977be085d7cf3"
server_key_iv: "93e6dad91872989ff42bf05f302ac156"
server_key_version: "encryptionkey@0"
I20260929 16:46:53.126154 21327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:46:53.141647 21327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:46:53.146551 21327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:46:53.146919 21327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:46:53.167272 21327 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40603
I20260929 16:46:53.167312 21393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40603 every 8 connection(s)
I20260929 16:46:53.168876 21327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:46:53.173281 21394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:46:53.177018 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 21327
I20260929 16:46:53.177245 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790700068255611-789-0/minicluster-data/master-0/wal/instance
I20260929 16:46:53.177780 789 external_mini_cluster.cc:1428] Setting key fe0eb0b8287fb6e22ce35d94227756d9
I20260929 16:46:53.187340 21394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c911fd713d14715b77e2793aac3e3a4: Bootstrap starting.
I20260929 16:46:53.191638 21394 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c911fd713d14715b77e2793aac3e3a4: Neither blocks nor log segments found. Creating new log.
I20260929 16:46:53.192955 21394 log.cc:824] T 00000000000000000000000000000000 P 8c911fd713d14715b77e2793aac3e3a4: Log is configured to *not* fsync() on all Append() calls
I20260929 16:46:53.193257 21397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:53.179982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49735 (local address 127.0.0.1:40603)
0929 16:46:53.180851 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:53.180864 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:53.180908 (+ 44us) server_negotiation.cc:407] Connection header received
0929 16:46:53.182752 (+ 1844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:53.182780 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:53.183378 (+ 598us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:53.183751 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:53.184423 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:53.185830 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:53.187874 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:53.188650 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:53.188812 (+ 162us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:53.190525 (+ 1713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:53.190533 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:53.190550 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:46:53.191259 (+ 709us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:53.191354 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:46:53.191382 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:53.191537 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:53.191964 (+ 427us) server_negotiation.cc:299] Negotiation successful
0929 16:46:53.192157 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":412,"threads_started":1}
I20260929 16:46:53.196358 21394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c911fd713d14715b77e2793aac3e3a4: No bootstrap required, opened a new log
I20260929 16:46:53.197950 789 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:46:53.205610 21394 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c911fd713d14715b77e2793aac3e3a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c911fd713d14715b77e2793aac3e3a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40603 } }
I20260929 16:46:53.206209 21394 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c911fd713d14715b77e2793aac3e3a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:46:53.206329 21394 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c911fd713d14715b77e2793aac3e3a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c911fd713d14715b77e2793aac3e3a4, State: Initialized, Role: FOLLOWER
I20260929 16:46:53.207173 21394 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c911fd713d14715b77e2793aac3e3a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c911fd713d14715b77e2793aac3e3a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40603 } }
I20260929 16:46:53.207465 21394 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c911fd713d14715b77e2793aac3e3a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:46:53.207652 21394 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c911fd713d14715b77e2793aac3e3a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:46:53.207850 21394 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c911fd713d14715b77e2793aac3e3a4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:46:53.210217 21397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:53.202785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35671 (local address 127.0.0.1:40603)
0929 16:46:53.203076 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:53.203082 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:53.203391 (+ 309us) server_negotiation.cc:407] Connection header received
0929 16:46:53.203602 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:53.203606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:53.203679 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:53.203786 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:53.204416 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:53.205194 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:53.207132 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:53.207776 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:53.207923 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:53.209248 (+ 1325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:53.209254 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:53.209258 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:46:53.209284 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:53.209328 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:46:53.209349 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:53.209452 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:53.209699 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 16:46:53.209785 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260929 16:46:53.211381 21394 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c911fd713d14715b77e2793aac3e3a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c911fd713d14715b77e2793aac3e3a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40603 } }
I20260929 16:46:53.212116 21394 leader_election.cc:302] T 00000000000000000000000000000000 P 8c911fd713d14715b77e2793aac3e3a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c911fd713d14715b77e2793aac3e3a4; no voters:
I20260929 16:46:53.212896 21394 leader_election.cc:288] T 00000000000000000000000000000000 P 8c911fd713d14715b77e2793aac3e3a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:46:53.213173 21404 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c911fd713d14715b77e2793aac3e3a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:46:53.213986 21404 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c911fd713d14715b77e2793aac3e3a4 [term 1 LEADER]: Becoming Leader. State: Replica: 8c911fd713d14715b77e2793aac3e3a4, State: Running, Role: LEADER
I20260929 16:46:53.214783 21404 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c911fd713d14715b77e2793aac3e3a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c911fd713d14715b77e2793aac3e3a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40603 } }
I20260929 16:46:53.215600 21394 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c911fd713d14715b77e2793aac3e3a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:46:53.218401 21405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c911fd713d14715b77e2793aac3e3a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c911fd713d14715b77e2793aac3e3a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c911fd713d14715b77e2793aac3e3a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40603 } } }
I20260929 16:46:53.219156 21406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c911fd713d14715b77e2793aac3e3a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c911fd713d14715b77e2793aac3e3a4. Latest consensus state: current_term: 1 leader_uuid: "8c911fd713d14715b77e2793aac3e3a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c911fd713d14715b77e2793aac3e3a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40603 } } }
I20260929 16:46:53.219548 21405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c911fd713d14715b77e2793aac3e3a4 [sys.catalog]: This master's current role is: LEADER
I20260929 16:46:53.219727 21406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c911fd713d14715b77e2793aac3e3a4 [sys.catalog]: This master's current role is: LEADER
I20260929 16:46:53.224624 21412 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:46:53.232465 21412 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:46:53.243379 21412 catalog_manager.cc:1347] Generated new cluster ID: f6f09227453c443ea8749cfc1fe58646
I20260929 16:46:53.243535 21412 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:46:53.265111 21412 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:46:53.267174 21412 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:46:53.280318 21412 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c911fd713d14715b77e2793aac3e3a4: Generated new TSK 0
I20260929 16:46:53.281417 21412 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:46:53.298573 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 21327
W20260929 16:46:53.299889 21389 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49735
W20260929 16:46:53.300702 789 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 21327: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:46:53.300871 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 21327
W20260929 16:46:53.318449 21312 connection.cc:535] client connection to 127.0.0.1:40603 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40603): Connection reset by peer (error 104)
2026-09-29T16:46:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2023 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790700068255611-789-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:46:53 dist-test-slave-l31f krb5kdc[21425](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:46:53 dist-test-slave-l31f krb5kdc[21425](info): set up 2 sockets
Sep 29 16:46:53 dist-test-slave-l31f krb5kdc[21425](info): commencing operation
krb5kdc: starting...
W20260929 16:46:54.447527 789 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.004s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790700068255611-789-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790700068255611-789-0/krb5kdc/test-user.keytab.
Sep 29 16:46:54 dist-test-slave-l31f krb5kdc[21425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:46:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:46:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790700068255611-789-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790700068255611-789-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:46:54.731345 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37127
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790700068255611-789-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37127
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:46:55.097886 21438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:46:55.098260 21438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:46:55.098454 21438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:46:55.106178 21438 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:46:55.106324 21438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:46:55.106423 21438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:46:55.106550 21438 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:46:55.106627 21438 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:46:55.117264 21438 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790700068255611-789-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37127
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790700068255611-789-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:46:55.118764 21438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:46:55.120388 21438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:46:55.135371 21443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:55.135417 21446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:55.135480 21444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:46:55.141392 21438 server_base.cc:1034] running on GCE node
I20260929 16:46:55.142621 21438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:46:55.144578 21438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:46:55.145853 21438 hybrid_clock.cc:648] HybridClock initialized: now 1790700415145800 us; error 47 us; skew 500 ppm
I20260929 16:46:55.146307 21438 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:46:55 dist-test-slave-l31f krb5kdc[21425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:46:55.150018 21438 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:46:55.152064 21438 webserver.cc:460] Webserver started at http://127.0.0.1:32879/ using document root <none> and password file <none>
I20260929 16:46:55.152889 21438 fs_manager.cc:359] Metadata directory not provided
I20260929 16:46:55.153064 21438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:46:55.153407 21438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:46:55.156934 21438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790700068255611-789-0/minicluster-data/master-0/data/instance:
uuid: "695dcdd558b441cfa7f95c4e4ba44c6a"
format_stamp: "Formatted at 2026-09-29 16:46:55 on dist-test-slave-l31f"
server_key: "36fc39fdf7820a5a55bd7dc9bd099245"
server_key_iv: "227e310079f31e02f60aed51b3e46fd1"
server_key_version: "encryptionkey@0"
I20260929 16:46:55.157876 21438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790700068255611-789-0/minicluster-data/master-0/wal/instance:
uuid: "695dcdd558b441cfa7f95c4e4ba44c6a"
format_stamp: "Formatted at 2026-09-29 16:46:55 on dist-test-slave-l31f"
server_key: "36fc39fdf7820a5a55bd7dc9bd099245"
server_key_iv: "227e310079f31e02f60aed51b3e46fd1"
server_key_version: "encryptionkey@0"
I20260929 16:46:55.164249 21438 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 16:46:55.168424 21453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:55.170053 21438 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:46:55.170302 21438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "695dcdd558b441cfa7f95c4e4ba44c6a"
format_stamp: "Formatted at 2026-09-29 16:46:55 on dist-test-slave-l31f"
server_key: "36fc39fdf7820a5a55bd7dc9bd099245"
server_key_iv: "227e310079f31e02f60aed51b3e46fd1"
server_key_version: "encryptionkey@0"
I20260929 16:46:55.170548 21438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:46:55.177305 21438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:46:55.182607 21438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:46:55.183069 21438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:46:55.204650 21438 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37127
I20260929 16:46:55.204689 21504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37127 every 8 connection(s)
I20260929 16:46:55.206388 21438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:46:55.210536 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 21438
I20260929 16:46:55.210748 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790700068255611-789-0/minicluster-data/master-0/wal/instance
I20260929 16:46:55.210878 21505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:46:55.211345 789 external_mini_cluster.cc:1428] Setting key 1cd613d7dda820707f9757e39723b86f
I20260929 16:46:55.220382 21508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:55.213265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44889 (local address 127.0.0.1:37127)
0929 16:46:55.214295 (+ 1030us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:55.214309 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:55.214359 (+ 50us) server_negotiation.cc:407] Connection header received
0929 16:46:55.216230 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:55.216259 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:55.216718 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:55.217151 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:55.218358 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:55.218365 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:55.218385 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:46:55.218405 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:55.218505 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:46:55.218534 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:55.218747 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:55.218877 (+ 130us) server_negotiation.cc:299] Negotiation successful
0929 16:46:55.219193 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":758,"thread_start_us":485,"threads_started":1}
I20260929 16:46:55.224707 789 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:46:55.226295 21505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 695dcdd558b441cfa7f95c4e4ba44c6a: Bootstrap starting.
I20260929 16:46:55.230548 21505 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 695dcdd558b441cfa7f95c4e4ba44c6a: Neither blocks nor log segments found. Creating new log.
I20260929 16:46:55.231818 21505 log.cc:824] T 00000000000000000000000000000000 P 695dcdd558b441cfa7f95c4e4ba44c6a: Log is configured to *not* fsync() on all Append() calls
I20260929 16:46:55.232980 21508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:55.229777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49663 (local address 127.0.0.1:37127)
0929 16:46:55.230054 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:55.230063 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:55.230376 (+ 313us) server_negotiation.cc:407] Connection header received
0929 16:46:55.230619 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:55.230632 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:55.230719 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:55.230830 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:55.231936 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:55.231942 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:55.231957 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:46:55.231981 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:55.232040 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:46:55.232064 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:55.232210 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:55.232354 (+ 144us) server_negotiation.cc:299] Negotiation successful
0929 16:46:55.232427 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:46:55.235627 21505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 695dcdd558b441cfa7f95c4e4ba44c6a: No bootstrap required, opened a new log
I20260929 16:46:55.243956 21505 raft_consensus.cc:348] T 00000000000000000000000000000000 P 695dcdd558b441cfa7f95c4e4ba44c6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695dcdd558b441cfa7f95c4e4ba44c6a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37127 } }
I20260929 16:46:55.244390 21505 raft_consensus.cc:374] T 00000000000000000000000000000000 P 695dcdd558b441cfa7f95c4e4ba44c6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:46:55.244524 21505 raft_consensus.cc:729] T 00000000000000000000000000000000 P 695dcdd558b441cfa7f95c4e4ba44c6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 695dcdd558b441cfa7f95c4e4ba44c6a, State: Initialized, Role: FOLLOWER
I20260929 16:46:55.245242 21505 consensus_queue.cc:260] T 00000000000000000000000000000000 P 695dcdd558b441cfa7f95c4e4ba44c6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695dcdd558b441cfa7f95c4e4ba44c6a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37127 } }
I20260929 16:46:55.245559 21505 raft_consensus.cc:388] T 00000000000000000000000000000000 P 695dcdd558b441cfa7f95c4e4ba44c6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:46:55.245740 21505 raft_consensus.cc:482] T 00000000000000000000000000000000 P 695dcdd558b441cfa7f95c4e4ba44c6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:46:55.245939 21505 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 695dcdd558b441cfa7f95c4e4ba44c6a [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:46:55.248347 21505 raft_consensus.cc:504] T 00000000000000000000000000000000 P 695dcdd558b441cfa7f95c4e4ba44c6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695dcdd558b441cfa7f95c4e4ba44c6a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37127 } }
I20260929 16:46:55.248929 21505 leader_election.cc:302] T 00000000000000000000000000000000 P 695dcdd558b441cfa7f95c4e4ba44c6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 695dcdd558b441cfa7f95c4e4ba44c6a; no voters:
I20260929 16:46:55.249598 21505 leader_election.cc:288] T 00000000000000000000000000000000 P 695dcdd558b441cfa7f95c4e4ba44c6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:46:55.249770 21515 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 695dcdd558b441cfa7f95c4e4ba44c6a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:46:55.250486 21515 raft_consensus.cc:686] T 00000000000000000000000000000000 P 695dcdd558b441cfa7f95c4e4ba44c6a [term 1 LEADER]: Becoming Leader. State: Replica: 695dcdd558b441cfa7f95c4e4ba44c6a, State: Running, Role: LEADER
I20260929 16:46:55.251093 21515 consensus_queue.cc:237] T 00000000000000000000000000000000 P 695dcdd558b441cfa7f95c4e4ba44c6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695dcdd558b441cfa7f95c4e4ba44c6a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37127 } }
I20260929 16:46:55.251904 21505 sys_catalog.cc:564] T 00000000000000000000000000000000 P 695dcdd558b441cfa7f95c4e4ba44c6a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:46:55.254487 21516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 695dcdd558b441cfa7f95c4e4ba44c6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "695dcdd558b441cfa7f95c4e4ba44c6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695dcdd558b441cfa7f95c4e4ba44c6a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37127 } } }
I20260929 16:46:55.255095 21516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 695dcdd558b441cfa7f95c4e4ba44c6a [sys.catalog]: This master's current role is: LEADER
I20260929 16:46:55.254835 21517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 695dcdd558b441cfa7f95c4e4ba44c6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 695dcdd558b441cfa7f95c4e4ba44c6a. Latest consensus state: current_term: 1 leader_uuid: "695dcdd558b441cfa7f95c4e4ba44c6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695dcdd558b441cfa7f95c4e4ba44c6a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37127 } } }
I20260929 16:46:55.255307 21517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 695dcdd558b441cfa7f95c4e4ba44c6a [sys.catalog]: This master's current role is: LEADER
I20260929 16:46:55.258661 21521 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:46:55.264889 21521 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:46:55.275570 21521 catalog_manager.cc:1347] Generated new cluster ID: f31bcb96b7664cf2a0af2a8ce7c23eb8
I20260929 16:46:55.275745 21521 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:46:55.322286 21521 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:46:55.322788 21521 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:46:55.334132 21521 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 695dcdd558b441cfa7f95c4e4ba44c6a: Generated new TSK 0
I20260929 16:46:55.335184 21521 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:46:55.349020 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 21438
W20260929 16:46:55.350232 21500 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44889
W20260929 16:46:55.351125 789 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 21438: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:46:55.351302 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 21438
2026-09-29T16:46:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2051 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790700068255611-789-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:46:55 dist-test-slave-l31f krb5kdc[21536](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:46:55 dist-test-slave-l31f krb5kdc[21536](info): set up 2 sockets
Sep 29 16:46:55 dist-test-slave-l31f krb5kdc[21536](info): commencing operation
krb5kdc: starting...
W20260929 16:46:56.487429 789 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790700068255611-789-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790700068255611-789-0/krb5kdc/test-user.keytab.
Sep 29 16:46:56 dist-test-slave-l31f krb5kdc[21536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:46:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:46:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790700068255611-789-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790700068255611-789-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:46:56.770781 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34957
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790700068255611-789-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:46:57.135685 21549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:46:57.136112 21549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:46:57.136284 21549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:46:57.144251 21549 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:46:57.144459 21549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:46:57.144567 21549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:46:57.144673 21549 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:46:57.144754 21549 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:46:57.155429 21549 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790700068255611-789-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790700068255611-789-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:46:57.156873 21549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:46:57.158272 21549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:46:57.171746 21554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:57.171772 21555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:57.171989 21557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:46:57.172741 21549 server_base.cc:1034] running on GCE node
I20260929 16:46:57.173641 21549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:46:57.175282 21549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:46:57.176517 21549 hybrid_clock.cc:648] HybridClock initialized: now 1790700417176477 us; error 37 us; skew 500 ppm
I20260929 16:46:57.176873 21549 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:46:57 dist-test-slave-l31f krb5kdc[21536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700417, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:46:57.180333 21549 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:46:57.182122 21549 webserver.cc:460] Webserver started at http://127.0.0.1:43905/ using document root <none> and password file <none>
I20260929 16:46:57.182933 21549 fs_manager.cc:359] Metadata directory not provided
I20260929 16:46:57.183144 21549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:46:57.183480 21549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:46:57.186739 21549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790700068255611-789-0/minicluster-data/master-0/data/instance:
uuid: "cce004a1b66c439bb5efef3bd07c04ea"
format_stamp: "Formatted at 2026-09-29 16:46:57 on dist-test-slave-l31f"
server_key: "969219e1e00a98638216bfea244a5b39"
server_key_iv: "7786f8429a473d0999ebab748a0f59c1"
server_key_version: "encryptionkey@0"
I20260929 16:46:57.187671 21549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790700068255611-789-0/minicluster-data/master-0/wal/instance:
uuid: "cce004a1b66c439bb5efef3bd07c04ea"
format_stamp: "Formatted at 2026-09-29 16:46:57 on dist-test-slave-l31f"
server_key: "969219e1e00a98638216bfea244a5b39"
server_key_iv: "7786f8429a473d0999ebab748a0f59c1"
server_key_version: "encryptionkey@0"
I20260929 16:46:57.193348 21549 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 16:46:57.197244 21564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:57.198774 21549 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:46:57.199039 21549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "cce004a1b66c439bb5efef3bd07c04ea"
format_stamp: "Formatted at 2026-09-29 16:46:57 on dist-test-slave-l31f"
server_key: "969219e1e00a98638216bfea244a5b39"
server_key_iv: "7786f8429a473d0999ebab748a0f59c1"
server_key_version: "encryptionkey@0"
I20260929 16:46:57.199306 21549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:46:57.221275 21549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:46:57.225968 21549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:46:57.226231 21549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:46:57.246443 21549 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34957
I20260929 16:46:57.246480 21615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34957 every 8 connection(s)
I20260929 16:46:57.248147 21549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:46:57.250047 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 21549
I20260929 16:46:57.250264 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790700068255611-789-0/minicluster-data/master-0/wal/instance
I20260929 16:46:57.250836 789 external_mini_cluster.cc:1428] Setting key bcb833cbca20b249a83c95c00e607113
I20260929 16:46:57.253780 21616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 16:46:57 dist-test-slave-l31f krb5kdc[21536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 16:46:57.268196 21616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cce004a1b66c439bb5efef3bd07c04ea: Bootstrap starting.
I20260929 16:46:57.272326 21616 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cce004a1b66c439bb5efef3bd07c04ea: Neither blocks nor log segments found. Creating new log.
I20260929 16:46:57.273494 21616 log.cc:824] T 00000000000000000000000000000000 P cce004a1b66c439bb5efef3bd07c04ea: Log is configured to *not* fsync() on all Append() calls
I20260929 16:46:57.277290 21616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cce004a1b66c439bb5efef3bd07c04ea: No bootstrap required, opened a new log
I20260929 16:46:57.277660 21619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:57.252897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40785 (local address 127.0.0.1:34957)
0929 16:46:57.253766 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:57.253785 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:57.253836 (+ 51us) server_negotiation.cc:407] Connection header received
0929 16:46:57.256298 (+ 2462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:57.256328 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:57.256896 (+ 568us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:57.257506 (+ 610us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:57.258823 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:57.261003 (+ 2180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:57.262675 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:57.263816 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:57.264033 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:57.268636 (+ 4603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:57.268647 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:57.268665 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:46:57.268711 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:57.272545 (+ 3834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:57.273456 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:57.273471 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:57.273482 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:57.273731 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:57.274224 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:57.274229 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:57.274232 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:57.274920 (+ 688us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:57.275277 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:57.275739 (+ 462us) server_negotiation.cc:299] Negotiation successful
0929 16:46:57.276239 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":374,"threads_started":1}
I20260929 16:46:57.282249 789 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:46:57.288581 21616 raft_consensus.cc:348] T 00000000000000000000000000000000 P cce004a1b66c439bb5efef3bd07c04ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cce004a1b66c439bb5efef3bd07c04ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34957 } }
I20260929 16:46:57.289155 21616 raft_consensus.cc:374] T 00000000000000000000000000000000 P cce004a1b66c439bb5efef3bd07c04ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:46:57.289275 21616 raft_consensus.cc:729] T 00000000000000000000000000000000 P cce004a1b66c439bb5efef3bd07c04ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cce004a1b66c439bb5efef3bd07c04ea, State: Initialized, Role: FOLLOWER
I20260929 16:46:57.289917 21616 consensus_queue.cc:260] T 00000000000000000000000000000000 P cce004a1b66c439bb5efef3bd07c04ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cce004a1b66c439bb5efef3bd07c04ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34957 } }
I20260929 16:46:57.290162 21616 raft_consensus.cc:388] T 00000000000000000000000000000000 P cce004a1b66c439bb5efef3bd07c04ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:46:57.290320 21616 raft_consensus.cc:482] T 00000000000000000000000000000000 P cce004a1b66c439bb5efef3bd07c04ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:46:57.290515 21616 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cce004a1b66c439bb5efef3bd07c04ea [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:46:57.293536 21616 raft_consensus.cc:504] T 00000000000000000000000000000000 P cce004a1b66c439bb5efef3bd07c04ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cce004a1b66c439bb5efef3bd07c04ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34957 } }
I20260929 16:46:57.294250 21616 leader_election.cc:302] T 00000000000000000000000000000000 P cce004a1b66c439bb5efef3bd07c04ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cce004a1b66c439bb5efef3bd07c04ea; no voters:
I20260929 16:46:57.295048 21616 leader_election.cc:288] T 00000000000000000000000000000000 P cce004a1b66c439bb5efef3bd07c04ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:46:57.295179 21626 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cce004a1b66c439bb5efef3bd07c04ea [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:46:57.296602 21626 raft_consensus.cc:686] T 00000000000000000000000000000000 P cce004a1b66c439bb5efef3bd07c04ea [term 1 LEADER]: Becoming Leader. State: Replica: cce004a1b66c439bb5efef3bd07c04ea, State: Running, Role: LEADER
I20260929 16:46:57.297260 21626 consensus_queue.cc:237] T 00000000000000000000000000000000 P cce004a1b66c439bb5efef3bd07c04ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cce004a1b66c439bb5efef3bd07c04ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34957 } }
I20260929 16:46:57.298856 21616 sys_catalog.cc:564] T 00000000000000000000000000000000 P cce004a1b66c439bb5efef3bd07c04ea [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:46:57.307281 21619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:57.288029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45537 (local address 127.0.0.1:34957)
0929 16:46:57.288245 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:57.288253 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:57.288727 (+ 474us) server_negotiation.cc:407] Connection header received
0929 16:46:57.288872 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:57.288877 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:57.288973 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:57.289100 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:57.291078 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:57.292246 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:57.293582 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:57.294269 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:57.294405 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:57.297779 (+ 3374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:57.297789 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:57.297797 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:46:57.297850 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:57.302632 (+ 4782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:57.304156 (+ 1524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:57.304164 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:57.304168 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:57.304395 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:46:57.305093 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:46:57.305098 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:46:57.305102 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:46:57.305411 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:57.305714 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:57.306135 (+ 421us) server_negotiation.cc:299] Negotiation successful
0929 16:46:57.306586 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 16:46:57.310649 21628 sys_catalog.cc:455] T 00000000000000000000000000000000 P cce004a1b66c439bb5efef3bd07c04ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader cce004a1b66c439bb5efef3bd07c04ea. Latest consensus state: current_term: 1 leader_uuid: "cce004a1b66c439bb5efef3bd07c04ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cce004a1b66c439bb5efef3bd07c04ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34957 } } }
I20260929 16:46:57.311177 21628 sys_catalog.cc:458] T 00000000000000000000000000000000 P cce004a1b66c439bb5efef3bd07c04ea [sys.catalog]: This master's current role is: LEADER
I20260929 16:46:57.311924 21627 sys_catalog.cc:455] T 00000000000000000000000000000000 P cce004a1b66c439bb5efef3bd07c04ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cce004a1b66c439bb5efef3bd07c04ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cce004a1b66c439bb5efef3bd07c04ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34957 } } }
I20260929 16:46:57.312505 21627 sys_catalog.cc:458] T 00000000000000000000000000000000 P cce004a1b66c439bb5efef3bd07c04ea [sys.catalog]: This master's current role is: LEADER
I20260929 16:46:57.312985 21640 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:46:57.319221 21640 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:46:57.332163 21640 catalog_manager.cc:1347] Generated new cluster ID: c2abd006b42f45bdb8c3ab97906afa40
I20260929 16:46:57.332409 21640 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:46:57.348374 21640 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:46:57.350775 21640 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:46:57.364389 21640 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cce004a1b66c439bb5efef3bd07c04ea: Generated new TSK 0
I20260929 16:46:57.365680 21640 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:46:57.377969 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 21549
I20260929 16:46:57.379838 21611 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:46:57.474546 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 21549
2026-09-29T16:46:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2123 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790700068255611-789-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:46:57 dist-test-slave-l31f krb5kdc[21648](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:46:57 dist-test-slave-l31f krb5kdc[21648](info): set up 2 sockets
Sep 29 16:46:57 dist-test-slave-l31f krb5kdc[21648](info): commencing operation
krb5kdc: starting...
W20260929 16:46:58.603631 789 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790700068255611-789-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790700068255611-789-0/krb5kdc/test-user.keytab.
Sep 29 16:46:58 dist-test-slave-l31f krb5kdc[21648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:46:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:46:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790700068255611-789-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790700068255611-789-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:46:58.889079 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44861
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790700068255611-789-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44861
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:46:59.246784 21661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:46:59.247294 21661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:46:59.247473 21661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:46:59.255447 21661 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:46:59.255586 21661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:46:59.255637 21661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:46:59.255743 21661 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:46:59.255823 21661 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:46:59.266515 21661 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790700068255611-789-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790700068255611-789-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:46:59.268265 21661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:46:59.269636 21661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:46:59.282546 21669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:59.282546 21667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:46:59.282894 21666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:46:59.283589 21661 server_base.cc:1034] running on GCE node
I20260929 16:46:59.284494 21661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:46:59.286190 21661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:46:59.287472 21661 hybrid_clock.cc:648] HybridClock initialized: now 1790700419287392 us; error 61 us; skew 500 ppm
I20260929 16:46:59.287919 21661 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:46:59 dist-test-slave-l31f krb5kdc[21648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:46:59.295521 21661 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:46:59.297463 21661 webserver.cc:460] Webserver started at http://127.0.0.1:44735/ using document root <none> and password file <none>
I20260929 16:46:59.298280 21661 fs_manager.cc:359] Metadata directory not provided
I20260929 16:46:59.298465 21661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:46:59.298830 21661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:46:59.302343 21661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790700068255611-789-0/minicluster-data/master-0/data/instance:
uuid: "a6f200f80aa2465d914cd1e12e8f07cd"
format_stamp: "Formatted at 2026-09-29 16:46:59 on dist-test-slave-l31f"
server_key: "18346c7af9ce2800d335147db52605a8"
server_key_iv: "9ad2182529dbbc33351e1422b0a3bc92"
server_key_version: "encryptionkey@0"
I20260929 16:46:59.303406 21661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790700068255611-789-0/minicluster-data/master-0/wal/instance:
uuid: "a6f200f80aa2465d914cd1e12e8f07cd"
format_stamp: "Formatted at 2026-09-29 16:46:59 on dist-test-slave-l31f"
server_key: "18346c7af9ce2800d335147db52605a8"
server_key_iv: "9ad2182529dbbc33351e1422b0a3bc92"
server_key_version: "encryptionkey@0"
I20260929 16:46:59.309651 21661 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 16:46:59.313657 21676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:46:59.315323 21661 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 16:46:59.315557 21661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "a6f200f80aa2465d914cd1e12e8f07cd"
format_stamp: "Formatted at 2026-09-29 16:46:59 on dist-test-slave-l31f"
server_key: "18346c7af9ce2800d335147db52605a8"
server_key_iv: "9ad2182529dbbc33351e1422b0a3bc92"
server_key_version: "encryptionkey@0"
I20260929 16:46:59.315793 21661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:46:59.333699 21661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:46:59.338825 21661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:46:59.339385 21661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:46:59.359939 21661 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44861
I20260929 16:46:59.359978 21727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44861 every 8 connection(s)
I20260929 16:46:59.361462 21661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:46:59.365513 21728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:46:59.367259 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 21661
I20260929 16:46:59.367467 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790700068255611-789-0/minicluster-data/master-0/wal/instance
I20260929 16:46:59.367995 789 external_mini_cluster.cc:1428] Setting key 321e4650d3e4022af91f3e579f0c2f82
I20260929 16:46:59.380045 21728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6f200f80aa2465d914cd1e12e8f07cd: Bootstrap starting.
I20260929 16:46:59.382535 21731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:59.370114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52377 (local address 127.0.0.1:44861)
0929 16:46:59.371162 (+ 1048us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:59.371180 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:59.371234 (+ 54us) server_negotiation.cc:407] Connection header received
0929 16:46:59.373023 (+ 1789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:59.373054 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:59.373591 (+ 537us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:59.374065 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:59.374679 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:59.376046 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:59.377467 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:59.378258 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:59.378496 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:59.379896 (+ 1400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:59.379904 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:59.379921 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:46:59.380628 (+ 707us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:59.380723 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:46:59.380752 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:59.380896 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:59.381156 (+ 260us) server_negotiation.cc:299] Negotiation successful
0929 16:46:59.381468 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":699,"thread_start_us":439,"threads_started":1}
I20260929 16:46:59.383706 21728 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6f200f80aa2465d914cd1e12e8f07cd: Neither blocks nor log segments found. Creating new log.
I20260929 16:46:59.384861 21728 log.cc:824] T 00000000000000000000000000000000 P a6f200f80aa2465d914cd1e12e8f07cd: Log is configured to *not* fsync() on all Append() calls
I20260929 16:46:59.386783 789 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:46:59.388366 21728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6f200f80aa2465d914cd1e12e8f07cd: No bootstrap required, opened a new log
I20260929 16:46:59.398306 21728 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6f200f80aa2465d914cd1e12e8f07cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6f200f80aa2465d914cd1e12e8f07cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44861 } }
I20260929 16:46:59.398937 21728 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6f200f80aa2465d914cd1e12e8f07cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:46:59.399128 21728 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6f200f80aa2465d914cd1e12e8f07cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6f200f80aa2465d914cd1e12e8f07cd, State: Initialized, Role: FOLLOWER
I20260929 16:46:59.399638 21731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:46:59.391419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58811 (local address 127.0.0.1:44861)
0929 16:46:59.391712 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0929 16:46:59.391721 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:46:59.392282 (+ 561us) server_negotiation.cc:407] Connection header received
0929 16:46:59.392440 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:46:59.392446 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:46:59.392538 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:46:59.392664 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:46:59.393227 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:59.394378 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:59.395921 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:46:59.396696 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:46:59.396851 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:46:59.398319 (+ 1468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:46:59.398326 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:46:59.398334 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:46:59.398362 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:46:59.398419 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:46:59.398442 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:46:59.398563 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 16:46:59.398711 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 16:46:59.398825 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260929 16:46:59.399914 21728 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6f200f80aa2465d914cd1e12e8f07cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6f200f80aa2465d914cd1e12e8f07cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44861 } }
I20260929 16:46:59.400208 21728 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6f200f80aa2465d914cd1e12e8f07cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:46:59.400374 21728 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6f200f80aa2465d914cd1e12e8f07cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:46:59.400574 21728 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6f200f80aa2465d914cd1e12e8f07cd [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:46:59.403126 21728 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6f200f80aa2465d914cd1e12e8f07cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6f200f80aa2465d914cd1e12e8f07cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44861 } }
I20260929 16:46:59.403738 21728 leader_election.cc:302] T 00000000000000000000000000000000 P a6f200f80aa2465d914cd1e12e8f07cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6f200f80aa2465d914cd1e12e8f07cd; no voters:
I20260929 16:46:59.404371 21728 leader_election.cc:288] T 00000000000000000000000000000000 P a6f200f80aa2465d914cd1e12e8f07cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:46:59.404536 21738 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6f200f80aa2465d914cd1e12e8f07cd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:46:59.405345 21738 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6f200f80aa2465d914cd1e12e8f07cd [term 1 LEADER]: Becoming Leader. State: Replica: a6f200f80aa2465d914cd1e12e8f07cd, State: Running, Role: LEADER
I20260929 16:46:59.405958 21738 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6f200f80aa2465d914cd1e12e8f07cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6f200f80aa2465d914cd1e12e8f07cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44861 } }
I20260929 16:46:59.406723 21728 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6f200f80aa2465d914cd1e12e8f07cd [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:46:59.409145 21739 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6f200f80aa2465d914cd1e12e8f07cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6f200f80aa2465d914cd1e12e8f07cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6f200f80aa2465d914cd1e12e8f07cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44861 } } }
I20260929 16:46:59.409569 21739 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6f200f80aa2465d914cd1e12e8f07cd [sys.catalog]: This master's current role is: LEADER
I20260929 16:46:59.409294 21740 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6f200f80aa2465d914cd1e12e8f07cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6f200f80aa2465d914cd1e12e8f07cd. Latest consensus state: current_term: 1 leader_uuid: "a6f200f80aa2465d914cd1e12e8f07cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6f200f80aa2465d914cd1e12e8f07cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44861 } } }
I20260929 16:46:59.409763 21740 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6f200f80aa2465d914cd1e12e8f07cd [sys.catalog]: This master's current role is: LEADER
I20260929 16:46:59.413439 21747 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:46:59.418876 21747 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:46:59.430264 21747 catalog_manager.cc:1347] Generated new cluster ID: f232f3d3e1b8454f98c2d3424aa63611
I20260929 16:46:59.430421 21747 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:46:59.444077 21747 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:46:59.445538 21747 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:46:59.460705 21747 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6f200f80aa2465d914cd1e12e8f07cd: Generated new TSK 0
I20260929 16:46:59.461704 21747 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:46:59.469525 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 21661
W20260929 16:46:59.470739 21723 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52377
W20260929 16:46:59.471562 789 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 21661: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:46:59.471726 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 21661
2026-09-29T16:46:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1996 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790700068255611-789-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:46:59 dist-test-slave-l31f krb5kdc[21759](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:46:59 dist-test-slave-l31f krb5kdc[21759](info): set up 2 sockets
Sep 29 16:46:59 dist-test-slave-l31f krb5kdc[21759](info): commencing operation
krb5kdc: starting...
W20260929 16:47:00.595683 789 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790700068255611-789-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790700068255611-789-0/krb5kdc/test-user.keytab.
Sep 29 16:47:00 dist-test-slave-l31f krb5kdc[21759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:47:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:47:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790700068255611-789-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790700068255611-789-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:47:00.879422 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34219
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790700068255611-789-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34219
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:47:01.235129 21772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:47:01.235476 21772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:47:01.235684 21772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:47:01.243654 21772 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:47:01.243822 21772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:47:01.243932 21772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:47:01.244230 21772 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:47:01.244315 21772 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:47:01.255462 21772 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790700068255611-789-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790700068255611-789-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:47:01.256896 21772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:47:01.258311 21772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:47:01.271109 21780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:47:01.271163 21778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:47:01.271142 21777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:47:01.272500 21772 server_base.cc:1034] running on GCE node
I20260929 16:47:01.273340 21772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:47:01.274947 21772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:47:01.276266 21772 hybrid_clock.cc:648] HybridClock initialized: now 1790700421276213 us; error 38 us; skew 500 ppm
I20260929 16:47:01.276698 21772 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:47:01 dist-test-slave-l31f krb5kdc[21759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700421, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:47:01.280190 21772 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:47:01.281962 21772 webserver.cc:460] Webserver started at http://127.0.0.1:46279/ using document root <none> and password file <none>
I20260929 16:47:01.282732 21772 fs_manager.cc:359] Metadata directory not provided
I20260929 16:47:01.282899 21772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:47:01.283330 21772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:47:01.286512 21772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790700068255611-789-0/minicluster-data/master-0/data/instance:
uuid: "42f56bcf3c0f4c4e8bc6df8530de13ba"
format_stamp: "Formatted at 2026-09-29 16:47:01 on dist-test-slave-l31f"
server_key: "d265bb15ba57ce58ddebc0e8e9f0e665"
server_key_iv: "5be62984322959a61daa1a1d0f3559f3"
server_key_version: "encryptionkey@0"
I20260929 16:47:01.287395 21772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790700068255611-789-0/minicluster-data/master-0/wal/instance:
uuid: "42f56bcf3c0f4c4e8bc6df8530de13ba"
format_stamp: "Formatted at 2026-09-29 16:47:01 on dist-test-slave-l31f"
server_key: "d265bb15ba57ce58ddebc0e8e9f0e665"
server_key_iv: "5be62984322959a61daa1a1d0f3559f3"
server_key_version: "encryptionkey@0"
I20260929 16:47:01.294240 21772 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260929 16:47:01.298336 21787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:47:01.300056 21772 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:47:01.300293 21772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "42f56bcf3c0f4c4e8bc6df8530de13ba"
format_stamp: "Formatted at 2026-09-29 16:47:01 on dist-test-slave-l31f"
server_key: "d265bb15ba57ce58ddebc0e8e9f0e665"
server_key_iv: "5be62984322959a61daa1a1d0f3559f3"
server_key_version: "encryptionkey@0"
I20260929 16:47:01.300542 21772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:47:01.307451 21772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:47:01.312732 21772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:47:01.313122 21772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:47:01.335425 21772 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34219
I20260929 16:47:01.335474 21838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34219 every 8 connection(s)
I20260929 16:47:01.337007 21772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:47:01.337445 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 21772
I20260929 16:47:01.337682 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790700068255611-789-0/minicluster-data/master-0/wal/instance
I20260929 16:47:01.338275 789 external_mini_cluster.cc:1428] Setting key f84f913f907de472f7c1eac2c3dacc4f
I20260929 16:47:01.342298 21839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:47:01.348352 21842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:01.340381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60821 (local address 127.0.0.1:34219)
0929 16:47:01.341249 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:01.341265 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:01.341318 (+ 53us) server_negotiation.cc:407] Connection header received
0929 16:47:01.343984 (+ 2666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:01.344019 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:01.344547 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:01.345058 (+ 511us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:01.346155 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:47:01.346163 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:47:01.346182 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:47:01.346199 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:47:01.346288 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:47:01.346321 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:47:01.346577 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:01.346736 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 16:47:01.346955 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":353,"threads_started":1}
I20260929 16:47:01.353096 789 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:47:01.357038 21839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42f56bcf3c0f4c4e8bc6df8530de13ba: Bootstrap starting.
I20260929 16:47:01.360641 21842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:01.357964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39877 (local address 127.0.0.1:34219)
0929 16:47:01.358172 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:01.358180 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:01.358585 (+ 405us) server_negotiation.cc:407] Connection header received
0929 16:47:01.358806 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:01.358810 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:01.358870 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:01.358994 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:01.359964 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:47:01.359972 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:47:01.359979 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:47:01.359998 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:47:01.360046 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:47:01.360062 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:47:01.360192 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:01.360288 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 16:47:01.360350 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:47:01.361194 21839 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42f56bcf3c0f4c4e8bc6df8530de13ba: Neither blocks nor log segments found. Creating new log.
I20260929 16:47:01.362488 21839 log.cc:824] T 00000000000000000000000000000000 P 42f56bcf3c0f4c4e8bc6df8530de13ba: Log is configured to *not* fsync() on all Append() calls
I20260929 16:47:01.365568 21839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42f56bcf3c0f4c4e8bc6df8530de13ba: No bootstrap required, opened a new log
I20260929 16:47:01.373509 21839 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42f56bcf3c0f4c4e8bc6df8530de13ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42f56bcf3c0f4c4e8bc6df8530de13ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34219 } }
I20260929 16:47:01.373925 21839 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42f56bcf3c0f4c4e8bc6df8530de13ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:47:01.374001 21839 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42f56bcf3c0f4c4e8bc6df8530de13ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42f56bcf3c0f4c4e8bc6df8530de13ba, State: Initialized, Role: FOLLOWER
I20260929 16:47:01.374595 21839 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42f56bcf3c0f4c4e8bc6df8530de13ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42f56bcf3c0f4c4e8bc6df8530de13ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34219 } }
I20260929 16:47:01.374804 21839 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42f56bcf3c0f4c4e8bc6df8530de13ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:47:01.374909 21839 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42f56bcf3c0f4c4e8bc6df8530de13ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:47:01.375118 21839 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42f56bcf3c0f4c4e8bc6df8530de13ba [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:47:01.377405 21839 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42f56bcf3c0f4c4e8bc6df8530de13ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42f56bcf3c0f4c4e8bc6df8530de13ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34219 } }
I20260929 16:47:01.377995 21839 leader_election.cc:302] T 00000000000000000000000000000000 P 42f56bcf3c0f4c4e8bc6df8530de13ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42f56bcf3c0f4c4e8bc6df8530de13ba; no voters:
I20260929 16:47:01.378718 21839 leader_election.cc:288] T 00000000000000000000000000000000 P 42f56bcf3c0f4c4e8bc6df8530de13ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:47:01.378849 21849 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42f56bcf3c0f4c4e8bc6df8530de13ba [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:47:01.379663 21849 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42f56bcf3c0f4c4e8bc6df8530de13ba [term 1 LEADER]: Becoming Leader. State: Replica: 42f56bcf3c0f4c4e8bc6df8530de13ba, State: Running, Role: LEADER
I20260929 16:47:01.380227 21849 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42f56bcf3c0f4c4e8bc6df8530de13ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42f56bcf3c0f4c4e8bc6df8530de13ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34219 } }
I20260929 16:47:01.381095 21839 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42f56bcf3c0f4c4e8bc6df8530de13ba [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:47:01.383526 21851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42f56bcf3c0f4c4e8bc6df8530de13ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42f56bcf3c0f4c4e8bc6df8530de13ba. Latest consensus state: current_term: 1 leader_uuid: "42f56bcf3c0f4c4e8bc6df8530de13ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42f56bcf3c0f4c4e8bc6df8530de13ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34219 } } }
I20260929 16:47:01.383520 21850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42f56bcf3c0f4c4e8bc6df8530de13ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42f56bcf3c0f4c4e8bc6df8530de13ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42f56bcf3c0f4c4e8bc6df8530de13ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34219 } } }
I20260929 16:47:01.384106 21851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42f56bcf3c0f4c4e8bc6df8530de13ba [sys.catalog]: This master's current role is: LEADER
I20260929 16:47:01.384114 21850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42f56bcf3c0f4c4e8bc6df8530de13ba [sys.catalog]: This master's current role is: LEADER
I20260929 16:47:01.385749 21854 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:47:01.392082 21854 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:47:01.403296 21854 catalog_manager.cc:1347] Generated new cluster ID: 6356fbcfce2d4ea6b73d5b9462476cfe
I20260929 16:47:01.403463 21854 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:47:01.427083 21854 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:47:01.427604 21854 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:47:01.436928 21854 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42f56bcf3c0f4c4e8bc6df8530de13ba: Generated new TSK 0
I20260929 16:47:01.437974 21854 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:47:01.451856 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 21772
W20260929 16:47:01.453438 21834 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60821
W20260929 16:47:01.454326 789 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 21772: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:47:01.454504 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 21772
2026-09-29T16:47:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1983 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790700068255611-789-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:47:01 dist-test-slave-l31f krb5kdc[21870](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:47:01 dist-test-slave-l31f krb5kdc[21870](info): set up 2 sockets
Sep 29 16:47:01 dist-test-slave-l31f krb5kdc[21870](info): commencing operation
krb5kdc: starting...
W20260929 16:47:02.587620 789 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.001s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790700068255611-789-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790700068255611-789-0/krb5kdc/test-user.keytab.
Sep 29 16:47:02 dist-test-slave-l31f krb5kdc[21870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:47:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:47:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790700068255611-789-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790700068255611-789-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:47:02.874526 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43691
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790700068255611-789-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:47:03.234565 21883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:47:03.235167 21883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:47:03.235330 21883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:47:03.243424 21883 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:47:03.243572 21883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:47:03.243621 21883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:47:03.243685 21883 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:47:03.243729 21883 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:47:03.254658 21883 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790700068255611-789-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790700068255611-789-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:47:03.256139 21883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:47:03.257501 21883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:47:03.271509 21888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:47:03.271602 21889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:47:03.271490 21891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:47:03.277463 21883 server_base.cc:1034] running on GCE node
I20260929 16:47:03.278471 21883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:47:03.280426 21883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:47:03.281706 21883 hybrid_clock.cc:648] HybridClock initialized: now 1790700423281662 us; error 35 us; skew 500 ppm
I20260929 16:47:03.282238 21883 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:47:03 dist-test-slave-l31f krb5kdc[21870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700423, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:47:03.286300 21883 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:47:03.288509 21883 webserver.cc:460] Webserver started at http://127.0.0.1:42263/ using document root <none> and password file <none>
I20260929 16:47:03.289391 21883 fs_manager.cc:359] Metadata directory not provided
I20260929 16:47:03.289518 21883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:47:03.290038 21883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:47:03.294003 21883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790700068255611-789-0/minicluster-data/master-0/data/instance:
uuid: "cb5f690ee80344d9ab0527d4439c41da"
format_stamp: "Formatted at 2026-09-29 16:47:03 on dist-test-slave-l31f"
server_key: "8f5368d0706e1a45c6063c27e4a145d6"
server_key_iv: "a2935d73b4fdcc24940d4787b34d8bc2"
server_key_version: "encryptionkey@0"
I20260929 16:47:03.295068 21883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790700068255611-789-0/minicluster-data/master-0/wal/instance:
uuid: "cb5f690ee80344d9ab0527d4439c41da"
format_stamp: "Formatted at 2026-09-29 16:47:03 on dist-test-slave-l31f"
server_key: "8f5368d0706e1a45c6063c27e4a145d6"
server_key_iv: "a2935d73b4fdcc24940d4787b34d8bc2"
server_key_version: "encryptionkey@0"
I20260929 16:47:03.301393 21883 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260929 16:47:03.306133 21898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:47:03.307963 21883 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 16:47:03.308200 21883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "cb5f690ee80344d9ab0527d4439c41da"
format_stamp: "Formatted at 2026-09-29 16:47:03 on dist-test-slave-l31f"
server_key: "8f5368d0706e1a45c6063c27e4a145d6"
server_key_iv: "a2935d73b4fdcc24940d4787b34d8bc2"
server_key_version: "encryptionkey@0"
I20260929 16:47:03.308418 21883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:47:03.330624 21883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:47:03.335850 21883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:47:03.336222 21883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:47:03.360623 21883 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43691
I20260929 16:47:03.360639 21949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43691 every 8 connection(s)
I20260929 16:47:03.362520 21883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:47:03.363874 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 21883
I20260929 16:47:03.364080 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790700068255611-789-0/minicluster-data/master-0/wal/instance
I20260929 16:47:03.364593 789 external_mini_cluster.cc:1428] Setting key a57942fa5a44306fec2c160dce8b6ffc
I20260929 16:47:03.367605 21950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 16:47:03 dist-test-slave-l31f krb5kdc[21870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 16:47:03.382180 21950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb5f690ee80344d9ab0527d4439c41da: Bootstrap starting.
I20260929 16:47:03.385941 21950 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb5f690ee80344d9ab0527d4439c41da: Neither blocks nor log segments found. Creating new log.
I20260929 16:47:03.387248 21950 log.cc:824] T 00000000000000000000000000000000 P cb5f690ee80344d9ab0527d4439c41da: Log is configured to *not* fsync() on all Append() calls
I20260929 16:47:03.391398 21950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb5f690ee80344d9ab0527d4439c41da: No bootstrap required, opened a new log
I20260929 16:47:03.393563 21953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:03.366595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.63:41919 (local address 127.0.0.1:43691)
0929 16:47:03.367509 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:03.367522 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:03.367571 (+ 49us) server_negotiation.cc:407] Connection header received
0929 16:47:03.369436 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:03.369460 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:03.369879 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:03.370262 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:03.372178 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:03.373627 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:03.375620 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:03.376480 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:03.376647 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:03.382709 (+ 6062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:47:03.382722 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:47:03.382747 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:47:03.382813 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:47:03.387341 (+ 4528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:03.388799 (+ 1458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:03.388810 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:03.388821 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:03.389075 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:03.389763 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:03.389768 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:03.389771 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:03.390581 (+ 810us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:47:03.390950 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:03.391638 (+ 688us) server_negotiation.cc:299] Negotiation successful
0929 16:47:03.392069 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":443,"threads_started":1}
I20260929 16:47:03.398658 789 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:47:03.401608 21950 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb5f690ee80344d9ab0527d4439c41da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5f690ee80344d9ab0527d4439c41da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43691 } }
I20260929 16:47:03.402241 21950 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb5f690ee80344d9ab0527d4439c41da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:47:03.402370 21950 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb5f690ee80344d9ab0527d4439c41da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb5f690ee80344d9ab0527d4439c41da, State: Initialized, Role: FOLLOWER
I20260929 16:47:03.403327 21950 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb5f690ee80344d9ab0527d4439c41da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5f690ee80344d9ab0527d4439c41da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43691 } }
I20260929 16:47:03.403651 21950 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb5f690ee80344d9ab0527d4439c41da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:47:03.403807 21950 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb5f690ee80344d9ab0527d4439c41da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:47:03.404031 21950 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb5f690ee80344d9ab0527d4439c41da [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:47:03.406862 21950 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb5f690ee80344d9ab0527d4439c41da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5f690ee80344d9ab0527d4439c41da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43691 } }
I20260929 16:47:03.407675 21950 leader_election.cc:302] T 00000000000000000000000000000000 P cb5f690ee80344d9ab0527d4439c41da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb5f690ee80344d9ab0527d4439c41da; no voters:
I20260929 16:47:03.408445 21950 leader_election.cc:288] T 00000000000000000000000000000000 P cb5f690ee80344d9ab0527d4439c41da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:47:03.408618 21960 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb5f690ee80344d9ab0527d4439c41da [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:47:03.410118 21960 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb5f690ee80344d9ab0527d4439c41da [term 1 LEADER]: Becoming Leader. State: Replica: cb5f690ee80344d9ab0527d4439c41da, State: Running, Role: LEADER
I20260929 16:47:03.410883 21960 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb5f690ee80344d9ab0527d4439c41da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5f690ee80344d9ab0527d4439c41da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43691 } }
I20260929 16:47:03.411285 21950 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb5f690ee80344d9ab0527d4439c41da [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:47:03.414170 21962 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb5f690ee80344d9ab0527d4439c41da [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb5f690ee80344d9ab0527d4439c41da. Latest consensus state: current_term: 1 leader_uuid: "cb5f690ee80344d9ab0527d4439c41da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5f690ee80344d9ab0527d4439c41da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43691 } } }
I20260929 16:47:03.414633 21962 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb5f690ee80344d9ab0527d4439c41da [sys.catalog]: This master's current role is: LEADER
I20260929 16:47:03.415148 21961 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb5f690ee80344d9ab0527d4439c41da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb5f690ee80344d9ab0527d4439c41da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5f690ee80344d9ab0527d4439c41da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43691 } } }
I20260929 16:47:03.415462 21961 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb5f690ee80344d9ab0527d4439c41da [sys.catalog]: This master's current role is: LEADER
I20260929 16:47:03.419914 21967 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:47:03.426110 21967 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:47:03.428429 21953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:03.404486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.63:37423 (local address 127.0.0.1:43691)
0929 16:47:03.404694 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:03.404701 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:03.405300 (+ 599us) server_negotiation.cc:407] Connection header received
0929 16:47:03.405457 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:03.405463 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:03.405557 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:03.405686 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:03.407401 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:03.408636 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:03.410609 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:03.411403 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:03.411555 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:03.416105 (+ 4550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:47:03.416117 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:47:03.416125 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:47:03.416193 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:47:03.421215 (+ 5022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:03.423655 (+ 2440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:03.423666 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:03.423674 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:03.424011 (+ 337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:03.424975 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:03.424986 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:03.424993 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:03.425614 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:47:03.426104 (+ 490us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:03.427380 (+ 1276us) server_negotiation.cc:299] Negotiation successful
0929 16:47:03.427720 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 16:47:03.441829 21967 catalog_manager.cc:1347] Generated new cluster ID: 2f6e8f7f613f4ba4b6b3982bd59b318a
I20260929 16:47:03.441968 21967 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:47:03.455350 21967 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:47:03.457101 21967 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:47:03.467303 21967 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb5f690ee80344d9ab0527d4439c41da: Generated new TSK 0
I20260929 16:47:03.468393 21967 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:47:03.483878 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 21883
I20260929 16:47:03.485919 21945 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:47:03.581490 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 21883
W20260929 16:47:03.599677 21868 connection.cc:535] client connection to 127.0.0.1:43691 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43691): Connection reset by peer (error 104)
2026-09-29T16:47:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2126 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790700068255611-789-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:47:03 dist-test-slave-l31f krb5kdc[21982](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:47:03 dist-test-slave-l31f krb5kdc[21982](info): set up 2 sockets
Sep 29 16:47:03 dist-test-slave-l31f krb5kdc[21982](info): commencing operation
krb5kdc: starting...
W20260929 16:47:04.743680 789 mini_kdc.cc:121] Time spent starting KDC: real 1.124s user 0.002s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790700068255611-789-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790700068255611-789-0/krb5kdc/test-user.keytab.
Sep 29 16:47:04 dist-test-slave-l31f krb5kdc[21982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:47:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:47:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790700068255611-789-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790700068255611-789-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:47:05.039068 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34959
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790700068255611-789-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:47:05.408617 21995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:47:05.409024 21995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:47:05.409212 21995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:47:05.417366 21995 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:47:05.417585 21995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:47:05.417698 21995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:47:05.417811 21995 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:47:05.417889 21995 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:47:05.428644 21995 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790700068255611-789-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790700068255611-789-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:47:05.430349 21995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:47:05.431970 21995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:47:05.447381 22000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:47:05.447317 22001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:47:05.447343 22003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:47:05.448012 21995 server_base.cc:1034] running on GCE node
I20260929 16:47:05.449095 21995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:47:05.450887 21995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:47:05.452170 21995 hybrid_clock.cc:648] HybridClock initialized: now 1790700425452087 us; error 79 us; skew 500 ppm
I20260929 16:47:05.452864 21995 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:47:05 dist-test-slave-l31f krb5kdc[21982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700425, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:47:05.463538 21995 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:47:05.465636 21995 webserver.cc:460] Webserver started at http://127.0.0.1:34119/ using document root <none> and password file <none>
I20260929 16:47:05.466549 21995 fs_manager.cc:359] Metadata directory not provided
I20260929 16:47:05.466682 21995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:47:05.467123 21995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:47:05.471316 21995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790700068255611-789-0/minicluster-data/master-0/data/instance:
uuid: "a029c4e36f75418eab4b9c0d4dd55db6"
format_stamp: "Formatted at 2026-09-29 16:47:05 on dist-test-slave-l31f"
server_key: "43c5cb568c066c848c9a7b7095947c40"
server_key_iv: "452ca81dab992ac35cba34c189e90c95"
server_key_version: "encryptionkey@0"
I20260929 16:47:05.472355 21995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790700068255611-789-0/minicluster-data/master-0/wal/instance:
uuid: "a029c4e36f75418eab4b9c0d4dd55db6"
format_stamp: "Formatted at 2026-09-29 16:47:05 on dist-test-slave-l31f"
server_key: "43c5cb568c066c848c9a7b7095947c40"
server_key_iv: "452ca81dab992ac35cba34c189e90c95"
server_key_version: "encryptionkey@0"
I20260929 16:47:05.479900 21995 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260929 16:47:05.484802 22010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:47:05.486634 21995 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:47:05.486886 21995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "a029c4e36f75418eab4b9c0d4dd55db6"
format_stamp: "Formatted at 2026-09-29 16:47:05 on dist-test-slave-l31f"
server_key: "43c5cb568c066c848c9a7b7095947c40"
server_key_iv: "452ca81dab992ac35cba34c189e90c95"
server_key_version: "encryptionkey@0"
I20260929 16:47:05.487192 21995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:47:05.517935 21995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:47:05.522538 21995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:47:05.522941 21995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:47:05.543774 21995 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34959
I20260929 16:47:05.543797 22061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34959 every 8 connection(s)
I20260929 16:47:05.545344 21995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:47:05.549191 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 21995
I20260929 16:47:05.549436 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790700068255611-789-0/minicluster-data/master-0/wal/instance
I20260929 16:47:05.549506 22062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:47:05.549924 789 external_mini_cluster.cc:1428] Setting key 69efe17ca62c46aea6b0515abfbe566a
I20260929 16:47:05.564818 22065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:05.551841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.63:37293 (local address 127.0.0.1:34959)
0929 16:47:05.552695 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:05.552710 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:05.552754 (+ 44us) server_negotiation.cc:407] Connection header received
0929 16:47:05.554615 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:05.554641 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:05.555237 (+ 596us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:05.555625 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:05.556304 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:05.557721 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:05.559645 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:05.560422 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:05.560588 (+ 166us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:05.562053 (+ 1465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:47:05.562061 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:47:05.562083 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:47:05.562773 (+ 690us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:47:05.562890 (+ 117us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:47:05.562920 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:47:05.563172 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:05.563521 (+ 349us) server_negotiation.cc:299] Negotiation successful
0929 16:47:05.563739 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":394,"threads_started":1}
I20260929 16:47:05.565102 22062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a029c4e36f75418eab4b9c0d4dd55db6: Bootstrap starting.
I20260929 16:47:05.569002 22062 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a029c4e36f75418eab4b9c0d4dd55db6: Neither blocks nor log segments found. Creating new log.
I20260929 16:47:05.569296 789 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:47:05.570183 22062 log.cc:824] T 00000000000000000000000000000000 P a029c4e36f75418eab4b9c0d4dd55db6: Log is configured to *not* fsync() on all Append() calls
I20260929 16:47:05.573503 22062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a029c4e36f75418eab4b9c0d4dd55db6: No bootstrap required, opened a new log
I20260929 16:47:05.581287 22065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:05.574077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.63:56623 (local address 127.0.0.1:34959)
0929 16:47:05.574277 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:05.574285 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:05.574767 (+ 482us) server_negotiation.cc:407] Connection header received
0929 16:47:05.574930 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:05.574936 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:05.575048 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:05.575170 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:05.575641 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:05.576858 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:05.578304 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:05.579129 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:05.579271 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:05.580430 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:47:05.580436 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:47:05.580442 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:47:05.580475 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:47:05.580549 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:47:05.580575 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:47:05.580696 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:05.580877 (+ 181us) server_negotiation.cc:299] Negotiation successful
0929 16:47:05.580942 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 16:47:05.583564 22062 raft_consensus.cc:348] T 00000000000000000000000000000000 P a029c4e36f75418eab4b9c0d4dd55db6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a029c4e36f75418eab4b9c0d4dd55db6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34959 } }
I20260929 16:47:05.584000 22062 raft_consensus.cc:374] T 00000000000000000000000000000000 P a029c4e36f75418eab4b9c0d4dd55db6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:47:05.584128 22062 raft_consensus.cc:729] T 00000000000000000000000000000000 P a029c4e36f75418eab4b9c0d4dd55db6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a029c4e36f75418eab4b9c0d4dd55db6, State: Initialized, Role: FOLLOWER
I20260929 16:47:05.584735 22062 consensus_queue.cc:260] T 00000000000000000000000000000000 P a029c4e36f75418eab4b9c0d4dd55db6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a029c4e36f75418eab4b9c0d4dd55db6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34959 } }
I20260929 16:47:05.584998 22062 raft_consensus.cc:388] T 00000000000000000000000000000000 P a029c4e36f75418eab4b9c0d4dd55db6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:47:05.585179 22062 raft_consensus.cc:482] T 00000000000000000000000000000000 P a029c4e36f75418eab4b9c0d4dd55db6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:47:05.585346 22062 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a029c4e36f75418eab4b9c0d4dd55db6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:47:05.587778 22062 raft_consensus.cc:504] T 00000000000000000000000000000000 P a029c4e36f75418eab4b9c0d4dd55db6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a029c4e36f75418eab4b9c0d4dd55db6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34959 } }
I20260929 16:47:05.588336 22062 leader_election.cc:302] T 00000000000000000000000000000000 P a029c4e36f75418eab4b9c0d4dd55db6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a029c4e36f75418eab4b9c0d4dd55db6; no voters:
I20260929 16:47:05.588996 22062 leader_election.cc:288] T 00000000000000000000000000000000 P a029c4e36f75418eab4b9c0d4dd55db6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:47:05.589190 22072 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a029c4e36f75418eab4b9c0d4dd55db6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:47:05.590262 22072 raft_consensus.cc:686] T 00000000000000000000000000000000 P a029c4e36f75418eab4b9c0d4dd55db6 [term 1 LEADER]: Becoming Leader. State: Replica: a029c4e36f75418eab4b9c0d4dd55db6, State: Running, Role: LEADER
I20260929 16:47:05.590848 22072 consensus_queue.cc:237] T 00000000000000000000000000000000 P a029c4e36f75418eab4b9c0d4dd55db6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a029c4e36f75418eab4b9c0d4dd55db6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34959 } }
I20260929 16:47:05.591439 22062 sys_catalog.cc:564] T 00000000000000000000000000000000 P a029c4e36f75418eab4b9c0d4dd55db6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:47:05.593863 22074 sys_catalog.cc:455] T 00000000000000000000000000000000 P a029c4e36f75418eab4b9c0d4dd55db6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a029c4e36f75418eab4b9c0d4dd55db6. Latest consensus state: current_term: 1 leader_uuid: "a029c4e36f75418eab4b9c0d4dd55db6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a029c4e36f75418eab4b9c0d4dd55db6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34959 } } }
I20260929 16:47:05.594269 22074 sys_catalog.cc:458] T 00000000000000000000000000000000 P a029c4e36f75418eab4b9c0d4dd55db6 [sys.catalog]: This master's current role is: LEADER
I20260929 16:47:05.594864 22073 sys_catalog.cc:455] T 00000000000000000000000000000000 P a029c4e36f75418eab4b9c0d4dd55db6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a029c4e36f75418eab4b9c0d4dd55db6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a029c4e36f75418eab4b9c0d4dd55db6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34959 } } }
I20260929 16:47:05.595325 22073 sys_catalog.cc:458] T 00000000000000000000000000000000 P a029c4e36f75418eab4b9c0d4dd55db6 [sys.catalog]: This master's current role is: LEADER
I20260929 16:47:05.600436 22078 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:47:05.606566 22078 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:47:05.616951 22078 catalog_manager.cc:1347] Generated new cluster ID: 9998c709c62148dc8a291d4d86354a17
I20260929 16:47:05.617100 22078 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:47:05.648325 22078 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:47:05.650435 22078 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:47:05.668788 22078 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a029c4e36f75418eab4b9c0d4dd55db6: Generated new TSK 0
I20260929 16:47:05.670002 22078 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:47:05.686744 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 21995
W20260929 16:47:05.688849 22057 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.63:37293
W20260929 16:47:05.689864 789 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 21995: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:47:05.690053 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 21995
W20260929 16:47:05.708855 21980 connection.cc:535] client connection to 127.0.0.1:34959 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34959): Connection reset by peer (error 104)
2026-09-29T16:47:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2109 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790700068255611-789-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:47:05 dist-test-slave-l31f krb5kdc[22094](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:47:05 dist-test-slave-l31f krb5kdc[22094](info): set up 2 sockets
Sep 29 16:47:05 dist-test-slave-l31f krb5kdc[22094](info): commencing operation
krb5kdc: starting...
W20260929 16:47:06.831724 789 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790700068255611-789-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790700068255611-789-0/krb5kdc/test-user.keytab.
Sep 29 16:47:07 dist-test-slave-l31f krb5kdc[22094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:47:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:47:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790700068255611-789-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790700068255611-789-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:47:07.119580 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37657
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790700068255611-789-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:47:07.490293 22107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:47:07.490712 22107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:47:07.490895 22107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:47:07.498669 22107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:47:07.498869 22107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:47:07.498996 22107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:47:07.499145 22107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:47:07.499225 22107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:47:07.509771 22107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790700068255611-789-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790700068255611-789-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:47:07.511299 22107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:47:07.512710 22107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:47:07.525930 22115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:47:07.525861 22113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:47:07.526230 22112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:47:07.526872 22107 server_base.cc:1034] running on GCE node
I20260929 16:47:07.527833 22107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:47:07.529452 22107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:47:07.530737 22107 hybrid_clock.cc:648] HybridClock initialized: now 1790700427530688 us; error 52 us; skew 500 ppm
I20260929 16:47:07.531210 22107 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:47:07 dist-test-slave-l31f krb5kdc[22094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700427, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:47:07.534731 22107 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:47:07.536520 22107 webserver.cc:460] Webserver started at http://127.0.0.1:37833/ using document root <none> and password file <none>
I20260929 16:47:07.537268 22107 fs_manager.cc:359] Metadata directory not provided
I20260929 16:47:07.537388 22107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:47:07.537683 22107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:47:07.541079 22107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790700068255611-789-0/minicluster-data/master-0/data/instance:
uuid: "b6047c886302465cb34ecb49f75d23e5"
format_stamp: "Formatted at 2026-09-29 16:47:07 on dist-test-slave-l31f"
server_key: "f32ac950935ae6e5f813b6cbf601d43a"
server_key_iv: "6740d8e2c3cb73e2ada6d6bf138ad117"
server_key_version: "encryptionkey@0"
I20260929 16:47:07.541944 22107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790700068255611-789-0/minicluster-data/master-0/wal/instance:
uuid: "b6047c886302465cb34ecb49f75d23e5"
format_stamp: "Formatted at 2026-09-29 16:47:07 on dist-test-slave-l31f"
server_key: "f32ac950935ae6e5f813b6cbf601d43a"
server_key_iv: "6740d8e2c3cb73e2ada6d6bf138ad117"
server_key_version: "encryptionkey@0"
I20260929 16:47:07.548023 22107 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 16:47:07.551941 22122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:47:07.553547 22107 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:47:07.553725 22107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "b6047c886302465cb34ecb49f75d23e5"
format_stamp: "Formatted at 2026-09-29 16:47:07 on dist-test-slave-l31f"
server_key: "f32ac950935ae6e5f813b6cbf601d43a"
server_key_iv: "6740d8e2c3cb73e2ada6d6bf138ad117"
server_key_version: "encryptionkey@0"
I20260929 16:47:07.553922 22107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:47:07.560346 22107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:47:07.565274 22107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:47:07.565562 22107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:47:07.585383 22107 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37657
I20260929 16:47:07.585436 22173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37657 every 8 connection(s)
I20260929 16:47:07.586930 22107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:47:07.590907 22174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:47:07.597126 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 22107
I20260929 16:47:07.597350 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790700068255611-789-0/minicluster-data/master-0/wal/instance
I20260929 16:47:07.597914 789 external_mini_cluster.cc:1428] Setting key d900e37ab970cccfd2399ce1dc2bfe10
I20260929 16:47:07.603302 22174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6047c886302465cb34ecb49f75d23e5: Bootstrap starting.
I20260929 16:47:07.607250 22177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:07.599998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.63:54609 (local address 127.0.0.1:37657)
0929 16:47:07.600931 (+ 933us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:07.600944 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:07.600994 (+ 50us) server_negotiation.cc:407] Connection header received
0929 16:47:07.602827 (+ 1833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:07.602875 (+ 48us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:07.603378 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:07.603802 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:07.605072 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:47:07.605079 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:47:07.605098 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:47:07.605117 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:47:07.605205 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:47:07.605230 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:47:07.605440 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:07.605561 (+ 121us) server_negotiation.cc:299] Negotiation successful
0929 16:47:07.605848 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":484,"threads_started":1}
I20260929 16:47:07.607889 22174 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6047c886302465cb34ecb49f75d23e5: Neither blocks nor log segments found. Creating new log.
I20260929 16:47:07.609531 22174 log.cc:824] T 00000000000000000000000000000000 P b6047c886302465cb34ecb49f75d23e5: Log is configured to *not* fsync() on all Append() calls
I20260929 16:47:07.612118 789 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:47:07.613188 22174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6047c886302465cb34ecb49f75d23e5: No bootstrap required, opened a new log
I20260929 16:47:07.620411 22177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:07.617480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.63:54251 (local address 127.0.0.1:37657)
0929 16:47:07.617713 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:07.617719 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:07.618069 (+ 350us) server_negotiation.cc:407] Connection header received
0929 16:47:07.618294 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:07.618302 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:07.618368 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:07.618467 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:07.619667 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:47:07.619672 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:47:07.619676 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:47:07.619696 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:47:07.619742 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:47:07.619761 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:47:07.619915 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:07.620003 (+ 88us) server_negotiation.cc:299] Negotiation successful
0929 16:47:07.620063 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260929 16:47:07.623731 22174 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6047c886302465cb34ecb49f75d23e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6047c886302465cb34ecb49f75d23e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37657 } }
I20260929 16:47:07.624207 22174 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6047c886302465cb34ecb49f75d23e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:47:07.624544 22174 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6047c886302465cb34ecb49f75d23e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6047c886302465cb34ecb49f75d23e5, State: Initialized, Role: FOLLOWER
I20260929 16:47:07.625209 22174 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6047c886302465cb34ecb49f75d23e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6047c886302465cb34ecb49f75d23e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37657 } }
I20260929 16:47:07.625490 22174 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6047c886302465cb34ecb49f75d23e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:47:07.625662 22174 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6047c886302465cb34ecb49f75d23e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:47:07.625849 22174 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6047c886302465cb34ecb49f75d23e5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:47:07.628904 22174 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6047c886302465cb34ecb49f75d23e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6047c886302465cb34ecb49f75d23e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37657 } }
I20260929 16:47:07.629565 22174 leader_election.cc:302] T 00000000000000000000000000000000 P b6047c886302465cb34ecb49f75d23e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6047c886302465cb34ecb49f75d23e5; no voters:
I20260929 16:47:07.630219 22174 leader_election.cc:288] T 00000000000000000000000000000000 P b6047c886302465cb34ecb49f75d23e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:47:07.630396 22184 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6047c886302465cb34ecb49f75d23e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:47:07.631206 22184 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6047c886302465cb34ecb49f75d23e5 [term 1 LEADER]: Becoming Leader. State: Replica: b6047c886302465cb34ecb49f75d23e5, State: Running, Role: LEADER
I20260929 16:47:07.631806 22184 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6047c886302465cb34ecb49f75d23e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6047c886302465cb34ecb49f75d23e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37657 } }
I20260929 16:47:07.632550 22174 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6047c886302465cb34ecb49f75d23e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:47:07.635126 22186 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6047c886302465cb34ecb49f75d23e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6047c886302465cb34ecb49f75d23e5. Latest consensus state: current_term: 1 leader_uuid: "b6047c886302465cb34ecb49f75d23e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6047c886302465cb34ecb49f75d23e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37657 } } }
I20260929 16:47:07.635627 22186 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6047c886302465cb34ecb49f75d23e5 [sys.catalog]: This master's current role is: LEADER
I20260929 16:47:07.635129 22185 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6047c886302465cb34ecb49f75d23e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6047c886302465cb34ecb49f75d23e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6047c886302465cb34ecb49f75d23e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37657 } } }
I20260929 16:47:07.636211 22185 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6047c886302465cb34ecb49f75d23e5 [sys.catalog]: This master's current role is: LEADER
I20260929 16:47:07.639533 22191 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:47:07.645704 22191 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:47:07.656738 22191 catalog_manager.cc:1347] Generated new cluster ID: 3ed5f2400c904627bd0f749e74944dc3
I20260929 16:47:07.656947 22191 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:47:07.683620 22191 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:47:07.684156 22191 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:47:07.696182 22191 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6047c886302465cb34ecb49f75d23e5: Generated new TSK 0
I20260929 16:47:07.697209 22191 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:47:07.706509 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 22107
W20260929 16:47:07.707885 22169 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.63:54609
W20260929 16:47:07.708683 789 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 22107: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:47:07.708819 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 22107
2026-09-29T16:47:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2018 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790700068255611-789-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:47:07 dist-test-slave-l31f krb5kdc[22205](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:47:07 dist-test-slave-l31f krb5kdc[22205](info): set up 2 sockets
Sep 29 16:47:07 dist-test-slave-l31f krb5kdc[22205](info): commencing operation
krb5kdc: starting...
W20260929 16:47:08.859647 789 mini_kdc.cc:121] Time spent starting KDC: real 1.113s user 0.001s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790700068255611-789-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790700068255611-789-0/krb5kdc/test-user.keytab.
Sep 29 16:47:09 dist-test-slave-l31f krb5kdc[22205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:47:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:47:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790700068255611-789-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790700068255611-789-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:47:09.147805 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45825
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790700068255611-789-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:47:09.502656 22218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:47:09.503120 22218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:47:09.503297 22218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:47:09.510901 22218 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:47:09.511142 22218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:47:09.511262 22218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:47:09.511384 22218 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:47:09.511474 22218 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:47:09.521875 22218 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790700068255611-789-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45825
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790700068255611-789-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:47:09.523411 22218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:47:09.524788 22218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:47:09.537664 22223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:47:09.537590 22224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:47:09.537627 22226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:47:09.539340 22218 server_base.cc:1034] running on GCE node
I20260929 16:47:09.540380 22218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:47:09.542719 22218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:47:09.544059 22218 hybrid_clock.cc:648] HybridClock initialized: now 1790700429544026 us; error 68 us; skew 500 ppm
I20260929 16:47:09.544492 22218 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:47:09 dist-test-slave-l31f krb5kdc[22205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700429, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:47:09.548003 22218 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:47:09.549746 22218 webserver.cc:460] Webserver started at http://127.0.0.1:45189/ using document root <none> and password file <none>
I20260929 16:47:09.550556 22218 fs_manager.cc:359] Metadata directory not provided
I20260929 16:47:09.550751 22218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:47:09.551183 22218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:47:09.554561 22218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790700068255611-789-0/minicluster-data/master-0/data/instance:
uuid: "1c99e11486874c9e8e0260cf34b8b142"
format_stamp: "Formatted at 2026-09-29 16:47:09 on dist-test-slave-l31f"
server_key: "c7ab152d11fea21b71bb06d77a0cea46"
server_key_iv: "05d38d2d424d8f24d9aa4907840dec0d"
server_key_version: "encryptionkey@0"
I20260929 16:47:09.555517 22218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790700068255611-789-0/minicluster-data/master-0/wal/instance:
uuid: "1c99e11486874c9e8e0260cf34b8b142"
format_stamp: "Formatted at 2026-09-29 16:47:09 on dist-test-slave-l31f"
server_key: "c7ab152d11fea21b71bb06d77a0cea46"
server_key_iv: "05d38d2d424d8f24d9aa4907840dec0d"
server_key_version: "encryptionkey@0"
I20260929 16:47:09.561517 22218 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.001s
I20260929 16:47:09.565675 22233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:47:09.567298 22218 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:47:09.567534 22218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "1c99e11486874c9e8e0260cf34b8b142"
format_stamp: "Formatted at 2026-09-29 16:47:09 on dist-test-slave-l31f"
server_key: "c7ab152d11fea21b71bb06d77a0cea46"
server_key_iv: "05d38d2d424d8f24d9aa4907840dec0d"
server_key_version: "encryptionkey@0"
I20260929 16:47:09.567773 22218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:47:09.585079 22218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:47:09.589812 22218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:47:09.590204 22218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:47:09.610347 22218 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45825
I20260929 16:47:09.610371 22284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45825 every 8 connection(s)
I20260929 16:47:09.611934 22218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:47:09.615799 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 22218
I20260929 16:47:09.616021 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790700068255611-789-0/minicluster-data/master-0/wal/instance
I20260929 16:47:09.616542 789 external_mini_cluster.cc:1428] Setting key ed813f073bd488315b912cfd5026c06c
I20260929 16:47:09.616582 22285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 16:47:09 dist-test-slave-l31f krb5kdc[22205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 16:47:09.632486 22285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c99e11486874c9e8e0260cf34b8b142: Bootstrap starting.
I20260929 16:47:09.636390 22285 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c99e11486874c9e8e0260cf34b8b142: Neither blocks nor log segments found. Creating new log.
I20260929 16:47:09.637624 22285 log.cc:824] T 00000000000000000000000000000000 P 1c99e11486874c9e8e0260cf34b8b142: Log is configured to *not* fsync() on all Append() calls
I20260929 16:47:09.640895 22285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c99e11486874c9e8e0260cf34b8b142: No bootstrap required, opened a new log
I20260929 16:47:09.642637 22288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:09.618480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.63:46815 (local address 127.0.0.1:45825)
0929 16:47:09.619368 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:09.619381 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:09.619430 (+ 49us) server_negotiation.cc:407] Connection header received
0929 16:47:09.621539 (+ 2109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:09.621567 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:09.622035 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:09.622411 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:09.624400 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:09.625763 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:09.627618 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:09.628372 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:09.628573 (+ 201us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:09.633839 (+ 5266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:47:09.633847 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:47:09.633865 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:47:09.633914 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:47:09.637501 (+ 3587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:09.638754 (+ 1253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:09.638763 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:09.638771 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:09.639051 (+ 280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:09.639600 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:09.639604 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:09.639606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:09.640166 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:47:09.640534 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:09.641093 (+ 559us) server_negotiation.cc:299] Negotiation successful
0929 16:47:09.641425 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":415,"threads_started":1}
I20260929 16:47:09.647181 789 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:47:09.649721 22285 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c99e11486874c9e8e0260cf34b8b142 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c99e11486874c9e8e0260cf34b8b142" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45825 } }
I20260929 16:47:09.650221 22285 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c99e11486874c9e8e0260cf34b8b142 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:47:09.650352 22285 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c99e11486874c9e8e0260cf34b8b142 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c99e11486874c9e8e0260cf34b8b142, State: Initialized, Role: FOLLOWER
I20260929 16:47:09.651129 22285 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c99e11486874c9e8e0260cf34b8b142 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c99e11486874c9e8e0260cf34b8b142" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45825 } }
I20260929 16:47:09.651436 22285 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c99e11486874c9e8e0260cf34b8b142 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:47:09.651626 22285 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c99e11486874c9e8e0260cf34b8b142 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:47:09.651855 22285 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c99e11486874c9e8e0260cf34b8b142 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:47:09.656072 22285 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c99e11486874c9e8e0260cf34b8b142 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c99e11486874c9e8e0260cf34b8b142" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45825 } }
I20260929 16:47:09.656740 22285 leader_election.cc:302] T 00000000000000000000000000000000 P 1c99e11486874c9e8e0260cf34b8b142 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c99e11486874c9e8e0260cf34b8b142; no voters:
I20260929 16:47:09.657485 22285 leader_election.cc:288] T 00000000000000000000000000000000 P 1c99e11486874c9e8e0260cf34b8b142 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:47:09.657625 22295 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c99e11486874c9e8e0260cf34b8b142 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:47:09.659315 22295 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c99e11486874c9e8e0260cf34b8b142 [term 1 LEADER]: Becoming Leader. State: Replica: 1c99e11486874c9e8e0260cf34b8b142, State: Running, Role: LEADER
I20260929 16:47:09.660055 22295 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c99e11486874c9e8e0260cf34b8b142 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c99e11486874c9e8e0260cf34b8b142" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45825 } }
I20260929 16:47:09.660475 22285 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c99e11486874c9e8e0260cf34b8b142 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:47:09.666728 22296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c99e11486874c9e8e0260cf34b8b142 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c99e11486874c9e8e0260cf34b8b142" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c99e11486874c9e8e0260cf34b8b142" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45825 } } }
I20260929 16:47:09.667302 22296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c99e11486874c9e8e0260cf34b8b142 [sys.catalog]: This master's current role is: LEADER
I20260929 16:47:09.667222 22297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c99e11486874c9e8e0260cf34b8b142 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c99e11486874c9e8e0260cf34b8b142. Latest consensus state: current_term: 1 leader_uuid: "1c99e11486874c9e8e0260cf34b8b142" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c99e11486874c9e8e0260cf34b8b142" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45825 } } }
I20260929 16:47:09.667518 22297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c99e11486874c9e8e0260cf34b8b142 [sys.catalog]: This master's current role is: LEADER
I20260929 16:47:09.670284 22301 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:47:09.675112 22288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:09.651895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.63:34951 (local address 127.0.0.1:45825)
0929 16:47:09.652077 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:09.652082 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:09.655653 (+ 3571us) server_negotiation.cc:407] Connection header received
0929 16:47:09.655882 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:09.655889 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:09.655978 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:09.656105 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:09.657781 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:09.659188 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:09.661136 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:09.661652 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:09.661797 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:09.664889 (+ 3092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:47:09.664899 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:47:09.664906 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:47:09.664968 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:47:09.669391 (+ 4423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:09.670806 (+ 1415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:09.670811 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:09.670816 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:09.671032 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:09.671857 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:09.671865 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:09.671871 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:09.672564 (+ 693us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:47:09.673172 (+ 608us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:09.674424 (+ 1252us) server_negotiation.cc:299] Negotiation successful
0929 16:47:09.674689 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 16:47:09.678020 22301 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:47:09.688453 22301 catalog_manager.cc:1347] Generated new cluster ID: bc081d842f05423183434bf5eef9485b
I20260929 16:47:09.688598 22301 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:47:09.710960 22301 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:47:09.712411 22301 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:47:09.721086 22301 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c99e11486874c9e8e0260cf34b8b142: Generated new TSK 0
I20260929 16:47:09.722046 22301 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:47:09.730685 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 22218
I20260929 16:47:09.732016 22280 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:47:09.827579 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 22218
W20260929 16:47:09.845721 22203 connection.cc:535] client connection to 127.0.0.1:45825 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45825): Connection reset by peer (error 104)
2026-09-29T16:47:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2119 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790700068255611-789-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:47:09 dist-test-slave-l31f krb5kdc[22317](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:47:09 dist-test-slave-l31f krb5kdc[22317](info): set up 2 sockets
Sep 29 16:47:09 dist-test-slave-l31f krb5kdc[22317](info): commencing operation
krb5kdc: starting...
W20260929 16:47:10.943738 789 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790700068255611-789-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790700068255611-789-0/krb5kdc/test-user.keytab.
Sep 29 16:47:11 dist-test-slave-l31f krb5kdc[22317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:47:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:47:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790700068255611-789-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790700068255611-789-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:47:11.231190 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35131
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790700068255611-789-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35131
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:47:11.584223 22330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:47:11.584622 22330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:47:11.584785 22330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:47:11.592417 22330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:47:11.592597 22330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:47:11.592712 22330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:47:11.592823 22330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:47:11.592900 22330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:47:11.603304 22330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790700068255611-789-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35131
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790700068255611-789-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:47:11.604774 22330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:47:11.606143 22330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:47:11.619239 22336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:47:11.619319 22338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:47:11.619216 22335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:47:11.625059 22330 server_base.cc:1034] running on GCE node
I20260929 16:47:11.626024 22330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:47:11.627795 22330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:47:11.629074 22330 hybrid_clock.cc:648] HybridClock initialized: now 1790700431629036 us; error 43 us; skew 500 ppm
I20260929 16:47:11.629495 22330 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:47:11 dist-test-slave-l31f krb5kdc[22317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:47:11.639122 22330 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:47:11.640879 22330 webserver.cc:460] Webserver started at http://127.0.0.1:43109/ using document root <none> and password file <none>
I20260929 16:47:11.641716 22330 fs_manager.cc:359] Metadata directory not provided
I20260929 16:47:11.641882 22330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:47:11.642253 22330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:47:11.645573 22330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790700068255611-789-0/minicluster-data/master-0/data/instance:
uuid: "54e4efa5690a4cbca822574e31108946"
format_stamp: "Formatted at 2026-09-29 16:47:11 on dist-test-slave-l31f"
server_key: "7211487266eaac8b3af1611da84f67b3"
server_key_iv: "9e005f1f64a1e864f1dc0be3461b0232"
server_key_version: "encryptionkey@0"
I20260929 16:47:11.646452 22330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790700068255611-789-0/minicluster-data/master-0/wal/instance:
uuid: "54e4efa5690a4cbca822574e31108946"
format_stamp: "Formatted at 2026-09-29 16:47:11 on dist-test-slave-l31f"
server_key: "7211487266eaac8b3af1611da84f67b3"
server_key_iv: "9e005f1f64a1e864f1dc0be3461b0232"
server_key_version: "encryptionkey@0"
I20260929 16:47:11.652529 22330 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260929 16:47:11.656598 22345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:47:11.658183 22330 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:47:11.658425 22330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "54e4efa5690a4cbca822574e31108946"
format_stamp: "Formatted at 2026-09-29 16:47:11 on dist-test-slave-l31f"
server_key: "7211487266eaac8b3af1611da84f67b3"
server_key_iv: "9e005f1f64a1e864f1dc0be3461b0232"
server_key_version: "encryptionkey@0"
I20260929 16:47:11.658669 22330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:47:11.684423 22330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:47:11.689391 22330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:47:11.689726 22330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:47:11.709910 22330 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35131
I20260929 16:47:11.709961 22396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35131 every 8 connection(s)
I20260929 16:47:11.711509 22330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:47:11.715670 22397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:47:11.719508 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 22330
I20260929 16:47:11.719717 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790700068255611-789-0/minicluster-data/master-0/wal/instance
I20260929 16:47:11.720259 789 external_mini_cluster.cc:1428] Setting key 583b62584cc086a110db4b3782654d99
I20260929 16:47:11.729276 22397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54e4efa5690a4cbca822574e31108946: Bootstrap starting.
I20260929 16:47:11.732905 22397 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54e4efa5690a4cbca822574e31108946: Neither blocks nor log segments found. Creating new log.
I20260929 16:47:11.734601 22397 log.cc:824] T 00000000000000000000000000000000 P 54e4efa5690a4cbca822574e31108946: Log is configured to *not* fsync() on all Append() calls
I20260929 16:47:11.735198 22400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:11.722386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.63:44371 (local address 127.0.0.1:35131)
0929 16:47:11.723362 (+ 976us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:11.723375 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:11.723423 (+ 48us) server_negotiation.cc:407] Connection header received
0929 16:47:11.725259 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:11.725289 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:11.725781 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:11.726246 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:11.726882 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:11.728350 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:11.729814 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:11.730566 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:11.730803 (+ 237us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:11.732172 (+ 1369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:47:11.732180 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:47:11.732198 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:47:11.732777 (+ 579us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:47:11.732867 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:47:11.732899 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:47:11.733094 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:11.733410 (+ 316us) server_negotiation.cc:299] Negotiation successful
0929 16:47:11.733659 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":741,"thread_start_us":421,"threads_started":1}
I20260929 16:47:11.738199 22397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54e4efa5690a4cbca822574e31108946: No bootstrap required, opened a new log
I20260929 16:47:11.739832 789 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:47:11.746977 22397 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54e4efa5690a4cbca822574e31108946 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54e4efa5690a4cbca822574e31108946" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35131 } }
I20260929 16:47:11.747539 22397 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54e4efa5690a4cbca822574e31108946 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:47:11.747671 22397 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54e4efa5690a4cbca822574e31108946 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54e4efa5690a4cbca822574e31108946, State: Initialized, Role: FOLLOWER
I20260929 16:47:11.748510 22397 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54e4efa5690a4cbca822574e31108946 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54e4efa5690a4cbca822574e31108946" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35131 } }
I20260929 16:47:11.748948 22397 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54e4efa5690a4cbca822574e31108946 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:47:11.749168 22397 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54e4efa5690a4cbca822574e31108946 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:47:11.749400 22397 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54e4efa5690a4cbca822574e31108946 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:47:11.751283 22400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:11.744524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.63:43065 (local address 127.0.0.1:35131)
0929 16:47:11.744753 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:11.744762 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:11.745068 (+ 306us) server_negotiation.cc:407] Connection header received
0929 16:47:11.745291 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:11.745295 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:11.745369 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:11.745503 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:11.746067 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:11.746827 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:11.748408 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:11.749019 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:11.749169 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:11.750469 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:47:11.750475 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:47:11.750479 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:47:11.750500 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:47:11.750547 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:47:11.750568 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:47:11.750712 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:11.750901 (+ 189us) server_negotiation.cc:299] Negotiation successful
0929 16:47:11.750976 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 16:47:11.752307 22397 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54e4efa5690a4cbca822574e31108946 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54e4efa5690a4cbca822574e31108946" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35131 } }
I20260929 16:47:11.752995 22397 leader_election.cc:302] T 00000000000000000000000000000000 P 54e4efa5690a4cbca822574e31108946 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54e4efa5690a4cbca822574e31108946; no voters:
I20260929 16:47:11.753804 22397 leader_election.cc:288] T 00000000000000000000000000000000 P 54e4efa5690a4cbca822574e31108946 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:47:11.753983 22407 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54e4efa5690a4cbca822574e31108946 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:47:11.755654 22407 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54e4efa5690a4cbca822574e31108946 [term 1 LEADER]: Becoming Leader. State: Replica: 54e4efa5690a4cbca822574e31108946, State: Running, Role: LEADER
I20260929 16:47:11.756367 22397 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54e4efa5690a4cbca822574e31108946 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:47:11.756316 22407 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54e4efa5690a4cbca822574e31108946 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54e4efa5690a4cbca822574e31108946" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35131 } }
I20260929 16:47:11.760707 22409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54e4efa5690a4cbca822574e31108946 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54e4efa5690a4cbca822574e31108946. Latest consensus state: current_term: 1 leader_uuid: "54e4efa5690a4cbca822574e31108946" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54e4efa5690a4cbca822574e31108946" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35131 } } }
I20260929 16:47:11.761106 22409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54e4efa5690a4cbca822574e31108946 [sys.catalog]: This master's current role is: LEADER
I20260929 16:47:11.762645 22415 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:47:11.762609 22408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54e4efa5690a4cbca822574e31108946 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54e4efa5690a4cbca822574e31108946" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54e4efa5690a4cbca822574e31108946" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35131 } } }
I20260929 16:47:11.763173 22408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54e4efa5690a4cbca822574e31108946 [sys.catalog]: This master's current role is: LEADER
I20260929 16:47:11.768791 22415 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:47:11.779587 22415 catalog_manager.cc:1347] Generated new cluster ID: 7108b6ae2f17462c9c0fdcec12658876
I20260929 16:47:11.779767 22415 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:47:11.797892 22415 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:47:11.799947 22415 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:47:11.807956 22415 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54e4efa5690a4cbca822574e31108946: Generated new TSK 0
I20260929 16:47:11.808938 22415 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:47:11.822609 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 22330
W20260929 16:47:11.823982 22392 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.63:44371
W20260929 16:47:11.825114 789 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 22330: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:47:11.825342 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 22330
W20260929 16:47:11.844380 22315 connection.cc:535] client connection to 127.0.0.1:35131 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35131): Connection reset by peer (error 104)
2026-09-29T16:47:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1998 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790700068255611-789-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:47:11 dist-test-slave-l31f krb5kdc[22428](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:47:11 dist-test-slave-l31f krb5kdc[22428](info): set up 2 sockets
Sep 29 16:47:11 dist-test-slave-l31f krb5kdc[22428](info): commencing operation
krb5kdc: starting...
W20260929 16:47:12.959687 789 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790700068255611-789-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790700068255611-789-0/krb5kdc/test-user.keytab.
Sep 29 16:47:13 dist-test-slave-l31f krb5kdc[22428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:47:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:47:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790700068255611-789-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790700068255611-789-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:47:13.242709 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43977
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790700068255611-789-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:47:13.593609 22441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:47:13.593933 22441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:47:13.594118 22441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:47:13.601708 22441 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:47:13.601845 22441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:47:13.601950 22441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:47:13.602074 22441 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:47:13.602147 22441 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:47:13.612543 22441 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790700068255611-789-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43977
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790700068255611-789-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:47:13.613998 22441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:47:13.615445 22441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:47:13.628263 22446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:47:13.628171 22449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:47:13.628214 22447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:47:13.629434 22441 server_base.cc:1034] running on GCE node
I20260929 16:47:13.630342 22441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:47:13.631996 22441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:47:13.633301 22441 hybrid_clock.cc:648] HybridClock initialized: now 1790700433633224 us; error 42 us; skew 500 ppm
I20260929 16:47:13.633924 22441 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:47:13 dist-test-slave-l31f krb5kdc[22428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:47:13.637715 22441 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:47:13.639511 22441 webserver.cc:460] Webserver started at http://127.0.0.1:41595/ using document root <none> and password file <none>
I20260929 16:47:13.640240 22441 fs_manager.cc:359] Metadata directory not provided
I20260929 16:47:13.640363 22441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:47:13.640642 22441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:47:13.644062 22441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790700068255611-789-0/minicluster-data/master-0/data/instance:
uuid: "a9051f92c815449ab3573f5ce5c9c5d8"
format_stamp: "Formatted at 2026-09-29 16:47:13 on dist-test-slave-l31f"
server_key: "ad1604bd0a86d839eab444df9f99b63b"
server_key_iv: "8a97720c5f100e4102547069c9c63e4e"
server_key_version: "encryptionkey@0"
I20260929 16:47:13.644882 22441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790700068255611-789-0/minicluster-data/master-0/wal/instance:
uuid: "a9051f92c815449ab3573f5ce5c9c5d8"
format_stamp: "Formatted at 2026-09-29 16:47:13 on dist-test-slave-l31f"
server_key: "ad1604bd0a86d839eab444df9f99b63b"
server_key_iv: "8a97720c5f100e4102547069c9c63e4e"
server_key_version: "encryptionkey@0"
I20260929 16:47:13.651124 22441 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 16:47:13.655166 22456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:47:13.656831 22441 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260929 16:47:13.657052 22441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "a9051f92c815449ab3573f5ce5c9c5d8"
format_stamp: "Formatted at 2026-09-29 16:47:13 on dist-test-slave-l31f"
server_key: "ad1604bd0a86d839eab444df9f99b63b"
server_key_iv: "8a97720c5f100e4102547069c9c63e4e"
server_key_version: "encryptionkey@0"
I20260929 16:47:13.657272 22441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:47:13.663800 22441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:47:13.669195 22441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:47:13.669584 22441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:47:13.690104 22441 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43977
I20260929 16:47:13.690157 22507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43977 every 8 connection(s)
I20260929 16:47:13.691689 22441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:47:13.695787 22508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:47:13.700122 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 22441
I20260929 16:47:13.700328 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790700068255611-789-0/minicluster-data/master-0/wal/instance
I20260929 16:47:13.700780 789 external_mini_cluster.cc:1428] Setting key 873c2e9720acf213c09e6ef5b5b39c11
I20260929 16:47:13.707823 22508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9051f92c815449ab3573f5ce5c9c5d8: Bootstrap starting.
I20260929 16:47:13.709712 22511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:13.702798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.63:34565 (local address 127.0.0.1:43977)
0929 16:47:13.703700 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:13.703712 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:13.703753 (+ 41us) server_negotiation.cc:407] Connection header received
0929 16:47:13.706027 (+ 2274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:13.706055 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:13.706448 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:13.706805 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:13.708073 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:47:13.708079 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:47:13.708094 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:47:13.708111 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:47:13.708175 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:47:13.708199 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:47:13.708378 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:13.708500 (+ 122us) server_negotiation.cc:299] Negotiation successful
0929 16:47:13.708727 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":429,"threads_started":1}
I20260929 16:47:13.711422 22508 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9051f92c815449ab3573f5ce5c9c5d8: Neither blocks nor log segments found. Creating new log.
I20260929 16:47:13.712767 22508 log.cc:824] T 00000000000000000000000000000000 P a9051f92c815449ab3573f5ce5c9c5d8: Log is configured to *not* fsync() on all Append() calls
I20260929 16:47:13.713706 789 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:47:13.715879 22508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9051f92c815449ab3573f5ce5c9c5d8: No bootstrap required, opened a new log
I20260929 16:47:13.720618 22511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:13.718082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.63:52971 (local address 127.0.0.1:43977)
0929 16:47:13.718287 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:13.718295 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:13.718693 (+ 398us) server_negotiation.cc:407] Connection header received
0929 16:47:13.718846 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:13.718852 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:13.718930 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:13.719075 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:13.719977 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:47:13.719983 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:47:13.719988 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:47:13.720007 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:47:13.720060 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:47:13.720082 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:47:13.720207 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:13.720257 (+ 50us) server_negotiation.cc:299] Negotiation successful
0929 16:47:13.720315 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 16:47:13.724740 22508 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9051f92c815449ab3573f5ce5c9c5d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9051f92c815449ab3573f5ce5c9c5d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43977 } }
I20260929 16:47:13.725186 22508 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9051f92c815449ab3573f5ce5c9c5d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:47:13.725306 22508 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9051f92c815449ab3573f5ce5c9c5d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9051f92c815449ab3573f5ce5c9c5d8, State: Initialized, Role: FOLLOWER
I20260929 16:47:13.725896 22508 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9051f92c815449ab3573f5ce5c9c5d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9051f92c815449ab3573f5ce5c9c5d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43977 } }
I20260929 16:47:13.726128 22508 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9051f92c815449ab3573f5ce5c9c5d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:47:13.726292 22508 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9051f92c815449ab3573f5ce5c9c5d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:47:13.726437 22508 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9051f92c815449ab3573f5ce5c9c5d8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:47:13.729148 22508 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9051f92c815449ab3573f5ce5c9c5d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9051f92c815449ab3573f5ce5c9c5d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43977 } }
I20260929 16:47:13.729635 22508 leader_election.cc:302] T 00000000000000000000000000000000 P a9051f92c815449ab3573f5ce5c9c5d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9051f92c815449ab3573f5ce5c9c5d8; no voters:
I20260929 16:47:13.730244 22508 leader_election.cc:288] T 00000000000000000000000000000000 P a9051f92c815449ab3573f5ce5c9c5d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:47:13.730435 22518 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9051f92c815449ab3573f5ce5c9c5d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:47:13.731269 22518 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9051f92c815449ab3573f5ce5c9c5d8 [term 1 LEADER]: Becoming Leader. State: Replica: a9051f92c815449ab3573f5ce5c9c5d8, State: Running, Role: LEADER
I20260929 16:47:13.731825 22518 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9051f92c815449ab3573f5ce5c9c5d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9051f92c815449ab3573f5ce5c9c5d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43977 } }
I20260929 16:47:13.732782 22508 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9051f92c815449ab3573f5ce5c9c5d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:47:13.737558 22519 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9051f92c815449ab3573f5ce5c9c5d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9051f92c815449ab3573f5ce5c9c5d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9051f92c815449ab3573f5ce5c9c5d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43977 } } }
I20260929 16:47:13.737983 22519 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9051f92c815449ab3573f5ce5c9c5d8 [sys.catalog]: This master's current role is: LEADER
I20260929 16:47:13.739126 22520 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9051f92c815449ab3573f5ce5c9c5d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9051f92c815449ab3573f5ce5c9c5d8. Latest consensus state: current_term: 1 leader_uuid: "a9051f92c815449ab3573f5ce5c9c5d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9051f92c815449ab3573f5ce5c9c5d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43977 } } }
I20260929 16:47:13.739518 22520 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9051f92c815449ab3573f5ce5c9c5d8 [sys.catalog]: This master's current role is: LEADER
I20260929 16:47:13.741784 22526 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:47:13.747676 22526 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:47:13.757519 22526 catalog_manager.cc:1347] Generated new cluster ID: 92faf9f377a3463a9b987cb8affcecbb
I20260929 16:47:13.757678 22526 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:47:13.782272 22526 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:47:13.782809 22526 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:47:13.794111 22526 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9051f92c815449ab3573f5ce5c9c5d8: Generated new TSK 0
I20260929 16:47:13.795179 22526 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:47:13.801518 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 22441
W20260929 16:47:13.803239 22503 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.63:34565
W20260929 16:47:13.804013 789 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 22441: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:47:13.804181 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 22441
2026-09-29T16:47:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1975 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790700068255611-789-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:47:13 dist-test-slave-l31f krb5kdc[22539](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:47:13 dist-test-slave-l31f krb5kdc[22539](info): set up 2 sockets
Sep 29 16:47:13 dist-test-slave-l31f krb5kdc[22539](info): commencing operation
krb5kdc: starting...
W20260929 16:47:14.923655 789 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790700068255611-789-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790700068255611-789-0/krb5kdc/test-user.keytab.
Sep 29 16:47:15 dist-test-slave-l31f krb5kdc[22539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:47:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:47:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.197.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
I20260929 16:47:15.211685 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:35145
--webserver_interface=127.0.197.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:39051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.126:35145
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:47:15.569595 22552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:47:15.569937 22552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:47:15.570117 22552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:47:15.577781 22552 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:47:15.577926 22552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:47:15.578024 22552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:47:15.578118 22552 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:47:15.578217 22552 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:47:15.589071 22552 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:39051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790700068255611-789-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.197.126:35145
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:35145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.197.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:47:15.590503 22552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:47:15.591915 22552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:47:15.605288 22558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:47:15.605386 22560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:47:15.605386 22557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:47:15.606485 22552 server_base.cc:1034] running on GCE node
I20260929 16:47:15.607605 22552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:47:15.609265 22552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:47:15.610535 22552 hybrid_clock.cc:648] HybridClock initialized: now 1790700435610475 us; error 44 us; skew 500 ppm
I20260929 16:47:15.611071 22552 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:47:15 dist-test-slave-l31f krb5kdc[22539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700435, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:47:15.614825 22552 init.cc:377] Logged in from keytab as kudu/127.0.197.126@KRBTEST.COM (short username kudu)
I20260929 16:47:15.616683 22552 webserver.cc:460] Webserver started at http://127.0.197.126:33537/ using document root <none> and password file <none>
I20260929 16:47:15.617509 22552 fs_manager.cc:359] Metadata directory not provided
I20260929 16:47:15.617658 22552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:47:15.618012 22552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:47:15.621502 22552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790700068255611-789-0/minicluster-data/master-0/data/instance:
uuid: "ec20a39cc7e447d3a2e7665e9a5f870f"
format_stamp: "Formatted at 2026-09-29 16:47:15 on dist-test-slave-l31f"
server_key: "13b7afdb8cce18582fc3f7013fcedac3"
server_key_iv: "3eda50684ab0adbb15789840003053a1"
server_key_version: "encryptionkey@0"
I20260929 16:47:15.622412 22552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790700068255611-789-0/minicluster-data/master-0/wal/instance:
uuid: "ec20a39cc7e447d3a2e7665e9a5f870f"
format_stamp: "Formatted at 2026-09-29 16:47:15 on dist-test-slave-l31f"
server_key: "13b7afdb8cce18582fc3f7013fcedac3"
server_key_iv: "3eda50684ab0adbb15789840003053a1"
server_key_version: "encryptionkey@0"
I20260929 16:47:15.628535 22552 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260929 16:47:15.632825 22567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:47:15.634649 22552 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:47:15.634924 22552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "ec20a39cc7e447d3a2e7665e9a5f870f"
format_stamp: "Formatted at 2026-09-29 16:47:15 on dist-test-slave-l31f"
server_key: "13b7afdb8cce18582fc3f7013fcedac3"
server_key_iv: "3eda50684ab0adbb15789840003053a1"
server_key_version: "encryptionkey@0"
I20260929 16:47:15.635206 22552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:47:15.653874 22552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:47:15.659245 22552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:47:15.659586 22552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:47:15.680225 22552 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.126:35145
I20260929 16:47:15.680248 22618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.126:35145 every 8 connection(s)
I20260929 16:47:15.681767 22552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:47:15.686072 22619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:47:15.690063 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 22552
I20260929 16:47:15.690286 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790700068255611-789-0/minicluster-data/master-0/wal/instance
I20260929 16:47:15.690724 789 external_mini_cluster.cc:1428] Setting key 399d85f1a6e4327205e9dd2b15e4f0e9
I20260929 16:47:15.699505 22619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec20a39cc7e447d3a2e7665e9a5f870f: Bootstrap starting.
I20260929 16:47:15.702894 22619 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec20a39cc7e447d3a2e7665e9a5f870f: Neither blocks nor log segments found. Creating new log.
I20260929 16:47:15.704454 22619 log.cc:824] T 00000000000000000000000000000000 P ec20a39cc7e447d3a2e7665e9a5f870f: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:47:15 dist-test-slave-l31f krb5kdc[22539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:47:15.708221 22619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec20a39cc7e447d3a2e7665e9a5f870f: No bootstrap required, opened a new log
I20260929 16:47:15.717294 22622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:15.692691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39599 (local address 127.0.197.126:35145)
0929 16:47:15.693657 (+ 966us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:15.693671 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:15.693718 (+ 47us) server_negotiation.cc:407] Connection header received
0929 16:47:15.695622 (+ 1904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:15.695651 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:15.696142 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:15.696628 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:15.698271 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:15.699819 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:15.701221 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:15.701955 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:15.702178 (+ 223us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:15.707728 (+ 5550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:47:15.707737 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:47:15.707753 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:47:15.707792 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:47:15.711582 (+ 3790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:15.712743 (+ 1161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:15.712751 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:15.712759 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:15.712999 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:15.713565 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:15.713569 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:15.713571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:15.714265 (+ 694us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:47:15.714530 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:15.715161 (+ 631us) server_negotiation.cc:299] Negotiation successful
0929 16:47:15.715611 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":380,"threads_started":1}
I20260929 16:47:15.718509 22619 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec20a39cc7e447d3a2e7665e9a5f870f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec20a39cc7e447d3a2e7665e9a5f870f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 35145 } }
I20260929 16:47:15.719065 22619 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec20a39cc7e447d3a2e7665e9a5f870f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:47:15.719200 22619 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec20a39cc7e447d3a2e7665e9a5f870f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec20a39cc7e447d3a2e7665e9a5f870f, State: Initialized, Role: FOLLOWER
I20260929 16:47:15.719852 22619 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec20a39cc7e447d3a2e7665e9a5f870f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec20a39cc7e447d3a2e7665e9a5f870f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 35145 } }
I20260929 16:47:15.720141 22619 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec20a39cc7e447d3a2e7665e9a5f870f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:47:15.720310 22619 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec20a39cc7e447d3a2e7665e9a5f870f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:47:15.720530 22619 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec20a39cc7e447d3a2e7665e9a5f870f [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:47:15.722146 789 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:47:15.723377 22619 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec20a39cc7e447d3a2e7665e9a5f870f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec20a39cc7e447d3a2e7665e9a5f870f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 35145 } }
I20260929 16:47:15.723965 22619 leader_election.cc:302] T 00000000000000000000000000000000 P ec20a39cc7e447d3a2e7665e9a5f870f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec20a39cc7e447d3a2e7665e9a5f870f; no voters:
I20260929 16:47:15.724601 22619 leader_election.cc:288] T 00000000000000000000000000000000 P ec20a39cc7e447d3a2e7665e9a5f870f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:47:15.724802 22624 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec20a39cc7e447d3a2e7665e9a5f870f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:47:15.726261 22624 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec20a39cc7e447d3a2e7665e9a5f870f [term 1 LEADER]: Becoming Leader. State: Replica: ec20a39cc7e447d3a2e7665e9a5f870f, State: Running, Role: LEADER
I20260929 16:47:15.727124 22624 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec20a39cc7e447d3a2e7665e9a5f870f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec20a39cc7e447d3a2e7665e9a5f870f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 35145 } }
I20260929 16:47:15.727383 22619 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec20a39cc7e447d3a2e7665e9a5f870f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:47:15.730841 22628 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec20a39cc7e447d3a2e7665e9a5f870f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec20a39cc7e447d3a2e7665e9a5f870f. Latest consensus state: current_term: 1 leader_uuid: "ec20a39cc7e447d3a2e7665e9a5f870f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec20a39cc7e447d3a2e7665e9a5f870f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 35145 } } }
I20260929 16:47:15.731386 22628 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec20a39cc7e447d3a2e7665e9a5f870f [sys.catalog]: This master's current role is: LEADER
I20260929 16:47:15.732110 22626 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec20a39cc7e447d3a2e7665e9a5f870f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec20a39cc7e447d3a2e7665e9a5f870f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec20a39cc7e447d3a2e7665e9a5f870f" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 35145 } } }
I20260929 16:47:15.732507 22626 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec20a39cc7e447d3a2e7665e9a5f870f [sys.catalog]: This master's current role is: LEADER
I20260929 16:47:15.736095 22635 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:47:15.742182 22635 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:47:15.751719 22622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:15.728701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58247 (local address 127.0.197.126:35145)
0929 16:47:15.728929 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:15.728939 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:15.731656 (+ 2717us) server_negotiation.cc:407] Connection header received
0929 16:47:15.731898 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:15.731904 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:15.731989 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:15.732094 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:15.734120 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:15.735522 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:15.737463 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:15.738257 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:15.738401 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:15.743444 (+ 5043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:47:15.743453 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:47:15.743459 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:47:15.743508 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:47:15.747805 (+ 4297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:15.748947 (+ 1142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:15.748959 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:15.748963 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:15.749143 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:15.749731 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:15.749738 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:15.749741 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:15.750200 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:47:15.750516 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:15.751071 (+ 555us) server_negotiation.cc:299] Negotiation successful
0929 16:47:15.751292 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 16:47:15.754006 22635 catalog_manager.cc:1347] Generated new cluster ID: d6de2ef09fba4d079220b3607bbeb786
I20260929 16:47:15.754156 22635 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:47:15.786772 22635 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:47:15.788223 22635 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:47:15.799203 22635 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec20a39cc7e447d3a2e7665e9a5f870f: Generated new TSK 0
I20260929 16:47:15.800196 22635 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:47:15.808808 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 22552
I20260929 16:47:15.810657 22614 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:47:15.907999 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 22552
W20260929 16:47:15.926518 22537 connection.cc:535] client connection to 127.0.197.126:35145 recv error: Network error: failed to read from TLS socket (remote: 127.0.197.126:35145): Connection reset by peer (error 104)
2026-09-29T16:47:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2106 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790700068255611-789-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:47:16 dist-test-slave-l31f krb5kdc[22651](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:47:16 dist-test-slave-l31f krb5kdc[22651](info): set up 2 sockets
Sep 29 16:47:16 dist-test-slave-l31f krb5kdc[22651](info): commencing operation
krb5kdc: starting...
W20260929 16:47:17.044153 789 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790700068255611-789-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790700068255611-789-0/krb5kdc/test-user.keytab.
Sep 29 16:47:17 dist-test-slave-l31f krb5kdc[22651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:47:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:47:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.197.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
I20260929 16:47:17.340444 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:46283
--webserver_interface=127.0.197.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:41017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.126:46283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:47:17.717496 22664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:47:17.717939 22664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:47:17.718120 22664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:47:17.726301 22664 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:47:17.726502 22664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:47:17.726606 22664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:47:17.726720 22664 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:47:17.726799 22664 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:47:17.737676 22664 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:41017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790700068255611-789-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.197.126:46283
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:46283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.197.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:47:17.739214 22664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:47:17.740617 22664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:47:17.753599 22669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:47:17.753680 22672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:47:17.753645 22670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:47:17.754655 22664 server_base.cc:1034] running on GCE node
I20260929 16:47:17.755595 22664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:47:17.757174 22664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:47:17.758425 22664 hybrid_clock.cc:648] HybridClock initialized: now 1790700437758377 us; error 58 us; skew 500 ppm
I20260929 16:47:17.758858 22664 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:47:17 dist-test-slave-l31f krb5kdc[22651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:47:17.762287 22664 init.cc:377] Logged in from keytab as kudu/127.0.197.126@KRBTEST.COM (short username kudu)
I20260929 16:47:17.764096 22664 webserver.cc:460] Webserver started at http://127.0.197.126:40377/ using document root <none> and password file <none>
I20260929 16:47:17.764868 22664 fs_manager.cc:359] Metadata directory not provided
I20260929 16:47:17.765038 22664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:47:17.765370 22664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:47:17.768657 22664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790700068255611-789-0/minicluster-data/master-0/data/instance:
uuid: "3afd250f27ec4c4ab01d862effb5b3c9"
format_stamp: "Formatted at 2026-09-29 16:47:17 on dist-test-slave-l31f"
server_key: "0fd7f77a6e2b4999606ec882b5c19771"
server_key_iv: "73d40dcf20541353037964656a18c2cd"
server_key_version: "encryptionkey@0"
I20260929 16:47:17.769502 22664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790700068255611-789-0/minicluster-data/master-0/wal/instance:
uuid: "3afd250f27ec4c4ab01d862effb5b3c9"
format_stamp: "Formatted at 2026-09-29 16:47:17 on dist-test-slave-l31f"
server_key: "0fd7f77a6e2b4999606ec882b5c19771"
server_key_iv: "73d40dcf20541353037964656a18c2cd"
server_key_version: "encryptionkey@0"
I20260929 16:47:17.775408 22664 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 16:47:17.779428 22679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:47:17.780987 22664 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:47:17.781210 22664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "3afd250f27ec4c4ab01d862effb5b3c9"
format_stamp: "Formatted at 2026-09-29 16:47:17 on dist-test-slave-l31f"
server_key: "0fd7f77a6e2b4999606ec882b5c19771"
server_key_iv: "73d40dcf20541353037964656a18c2cd"
server_key_version: "encryptionkey@0"
I20260929 16:47:17.781443 22664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:47:17.800063 22664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:47:17.804698 22664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:47:17.805034 22664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:47:17.826263 22664 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.126:46283
I20260929 16:47:17.826303 22730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.126:46283 every 8 connection(s)
I20260929 16:47:17.828380 22664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:47:17.830561 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 22664
I20260929 16:47:17.830785 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790700068255611-789-0/minicluster-data/master-0/wal/instance
I20260929 16:47:17.831375 789 external_mini_cluster.cc:1428] Setting key 25fddd50440163b34a44e2a89febbd5b
I20260929 16:47:17.834128 22731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:47:17.846403 22734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:17.833750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42781 (local address 127.0.197.126:46283)
0929 16:47:17.834767 (+ 1017us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:17.834780 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:17.834823 (+ 43us) server_negotiation.cc:407] Connection header received
0929 16:47:17.836628 (+ 1805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:17.836654 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:17.837114 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:17.837612 (+ 498us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:17.838233 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:17.839771 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:17.841234 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:17.842011 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:17.842250 (+ 239us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:17.843750 (+ 1500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:47:17.843758 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:47:17.843777 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:47:17.844418 (+ 641us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:47:17.844517 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:47:17.844550 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:47:17.844757 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:17.845061 (+ 304us) server_negotiation.cc:299] Negotiation successful
0929 16:47:17.845280 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":784,"thread_start_us":510,"threads_started":1}
I20260929 16:47:17.848107 22731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3afd250f27ec4c4ab01d862effb5b3c9: Bootstrap starting.
I20260929 16:47:17.850958 789 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:47:17.852161 22731 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3afd250f27ec4c4ab01d862effb5b3c9: Neither blocks nor log segments found. Creating new log.
I20260929 16:47:17.853425 22731 log.cc:824] T 00000000000000000000000000000000 P 3afd250f27ec4c4ab01d862effb5b3c9: Log is configured to *not* fsync() on all Append() calls
I20260929 16:47:17.857611 22731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3afd250f27ec4c4ab01d862effb5b3c9: No bootstrap required, opened a new log
I20260929 16:47:17.862879 22734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:17.856009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44395 (local address 127.0.197.126:46283)
0929 16:47:17.856220 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:17.856229 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:17.856619 (+ 390us) server_negotiation.cc:407] Connection header received
0929 16:47:17.856858 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:17.856863 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:17.856926 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:17.857040 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:17.857613 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:17.858482 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:17.860026 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:17.860537 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:17.860697 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:17.862029 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:47:17.862034 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:47:17.862038 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:47:17.862063 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:47:17.862109 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:47:17.862127 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:47:17.862272 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:17.862488 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 16:47:17.862552 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 16:47:17.868296 22731 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3afd250f27ec4c4ab01d862effb5b3c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3afd250f27ec4c4ab01d862effb5b3c9" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 46283 } }
I20260929 16:47:17.868826 22731 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3afd250f27ec4c4ab01d862effb5b3c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:47:17.868970 22731 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3afd250f27ec4c4ab01d862effb5b3c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3afd250f27ec4c4ab01d862effb5b3c9, State: Initialized, Role: FOLLOWER
I20260929 16:47:17.869632 22731 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3afd250f27ec4c4ab01d862effb5b3c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3afd250f27ec4c4ab01d862effb5b3c9" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 46283 } }
I20260929 16:47:17.869946 22731 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3afd250f27ec4c4ab01d862effb5b3c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:47:17.870108 22731 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3afd250f27ec4c4ab01d862effb5b3c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:47:17.870297 22731 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3afd250f27ec4c4ab01d862effb5b3c9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:47:17.873067 22731 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3afd250f27ec4c4ab01d862effb5b3c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3afd250f27ec4c4ab01d862effb5b3c9" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 46283 } }
I20260929 16:47:17.873589 22731 leader_election.cc:302] T 00000000000000000000000000000000 P 3afd250f27ec4c4ab01d862effb5b3c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3afd250f27ec4c4ab01d862effb5b3c9; no voters:
I20260929 16:47:17.874328 22731 leader_election.cc:288] T 00000000000000000000000000000000 P 3afd250f27ec4c4ab01d862effb5b3c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:47:17.874495 22741 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3afd250f27ec4c4ab01d862effb5b3c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:47:17.875626 22741 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3afd250f27ec4c4ab01d862effb5b3c9 [term 1 LEADER]: Becoming Leader. State: Replica: 3afd250f27ec4c4ab01d862effb5b3c9, State: Running, Role: LEADER
I20260929 16:47:17.876217 22741 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3afd250f27ec4c4ab01d862effb5b3c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3afd250f27ec4c4ab01d862effb5b3c9" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 46283 } }
I20260929 16:47:17.876998 22731 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3afd250f27ec4c4ab01d862effb5b3c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:47:17.879397 22742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3afd250f27ec4c4ab01d862effb5b3c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3afd250f27ec4c4ab01d862effb5b3c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3afd250f27ec4c4ab01d862effb5b3c9" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 46283 } } }
I20260929 16:47:17.879837 22742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3afd250f27ec4c4ab01d862effb5b3c9 [sys.catalog]: This master's current role is: LEADER
I20260929 16:47:17.882167 22743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3afd250f27ec4c4ab01d862effb5b3c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3afd250f27ec4c4ab01d862effb5b3c9. Latest consensus state: current_term: 1 leader_uuid: "3afd250f27ec4c4ab01d862effb5b3c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3afd250f27ec4c4ab01d862effb5b3c9" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 46283 } } }
I20260929 16:47:17.882737 22743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3afd250f27ec4c4ab01d862effb5b3c9 [sys.catalog]: This master's current role is: LEADER
I20260929 16:47:17.885234 22747 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:47:17.891105 22747 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:47:17.901362 22747 catalog_manager.cc:1347] Generated new cluster ID: 731a1c2f71b54a9d9fb0e66b3a8d7093
I20260929 16:47:17.901582 22747 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:47:17.935314 22747 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:47:17.936784 22747 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:47:17.947153 22747 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3afd250f27ec4c4ab01d862effb5b3c9: Generated new TSK 0
I20260929 16:47:17.948323 22747 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:47:17.966957 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 22664
W20260929 16:47:17.968178 22726 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42781
W20260929 16:47:17.968972 789 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 22664: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:47:17.969127 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 22664
W20260929 16:47:17.986804 22649 connection.cc:535] client connection to 127.0.197.126:46283 recv error: Network error: failed to read from TLS socket (remote: 127.0.197.126:46283): Connection reset by peer (error 104)
2026-09-29T16:47:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2059 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790700068255611-789-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:47:18 dist-test-slave-l31f krb5kdc[22762](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:47:18 dist-test-slave-l31f krb5kdc[22762](info): set up 2 sockets
Sep 29 16:47:18 dist-test-slave-l31f krb5kdc[22762](info): commencing operation
krb5kdc: starting...
W20260929 16:47:19.116622 789 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.008s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790700068255611-789-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790700068255611-789-0/krb5kdc/test-user.keytab.
Sep 29 16:47:19 dist-test-slave-l31f krb5kdc[22762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:47:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:47:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.197.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
I20260929 16:47:19.408418 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:43457
--webserver_interface=127.0.197.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:37585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.126:43457
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:47:19.760754 22775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:47:19.761089 22775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:47:19.761276 22775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:47:19.768891 22775 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:47:19.769040 22775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:47:19.769131 22775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:47:19.769229 22775 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:47:19.769294 22775 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:47:19.780436 22775 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:37585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790700068255611-789-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.197.126:43457
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:43457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.197.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:47:19.781847 22775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:47:19.783324 22775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:47:19.796897 22780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:47:19.796790 22781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:47:19.796866 22783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:47:19.797725 22775 server_base.cc:1034] running on GCE node
I20260929 16:47:19.798681 22775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:47:19.800380 22775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:47:19.801666 22775 hybrid_clock.cc:648] HybridClock initialized: now 1790700439801629 us; error 61 us; skew 500 ppm
I20260929 16:47:19.802099 22775 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:47:19 dist-test-slave-l31f krb5kdc[22762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:47:19.805989 22775 init.cc:377] Logged in from keytab as kudu/127.0.197.126@KRBTEST.COM (short username kudu)
I20260929 16:47:19.807744 22775 webserver.cc:460] Webserver started at http://127.0.197.126:42473/ using document root <none> and password file <none>
I20260929 16:47:19.808537 22775 fs_manager.cc:359] Metadata directory not provided
I20260929 16:47:19.808699 22775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:47:19.809041 22775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:47:19.812479 22775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790700068255611-789-0/minicluster-data/master-0/data/instance:
uuid: "c7e28d8f88e942f2a5a1c9d841220787"
format_stamp: "Formatted at 2026-09-29 16:47:19 on dist-test-slave-l31f"
server_key: "e9a7f205eab705b8d3b5579c9595aa5d"
server_key_iv: "7da90d15207527e61706a3d05b4df8d3"
server_key_version: "encryptionkey@0"
I20260929 16:47:19.813364 22775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790700068255611-789-0/minicluster-data/master-0/wal/instance:
uuid: "c7e28d8f88e942f2a5a1c9d841220787"
format_stamp: "Formatted at 2026-09-29 16:47:19 on dist-test-slave-l31f"
server_key: "e9a7f205eab705b8d3b5579c9595aa5d"
server_key_iv: "7da90d15207527e61706a3d05b4df8d3"
server_key_version: "encryptionkey@0"
I20260929 16:47:19.819854 22775 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260929 16:47:19.824103 22790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:47:19.825778 22775 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260929 16:47:19.826027 22775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "c7e28d8f88e942f2a5a1c9d841220787"
format_stamp: "Formatted at 2026-09-29 16:47:19 on dist-test-slave-l31f"
server_key: "e9a7f205eab705b8d3b5579c9595aa5d"
server_key_iv: "7da90d15207527e61706a3d05b4df8d3"
server_key_version: "encryptionkey@0"
I20260929 16:47:19.826289 22775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:47:19.833042 22775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:47:19.838840 22775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:47:19.839250 22775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:47:19.859550 22775 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.126:43457
I20260929 16:47:19.859552 22841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.126:43457 every 8 connection(s)
I20260929 16:47:19.861564 22775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:47:19.865931 22842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:47:19.866307 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 22775
I20260929 16:47:19.866504 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790700068255611-789-0/minicluster-data/master-0/wal/instance
I20260929 16:47:19.866959 789 external_mini_cluster.cc:1428] Setting key c38dd82fc09d2f92f99f7db6bfbf8077
I20260929 16:47:19.877549 22845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:19.869038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46185 (local address 127.0.197.126:43457)
0929 16:47:19.870017 (+ 979us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:19.870034 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:19.870090 (+ 56us) server_negotiation.cc:407] Connection header received
0929 16:47:19.873218 (+ 3128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:19.873259 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:19.873856 (+ 597us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:19.874389 (+ 533us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:19.875477 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:47:19.875485 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:47:19.875502 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:47:19.875532 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:47:19.875604 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:47:19.875635 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:47:19.875870 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:19.876001 (+ 131us) server_negotiation.cc:299] Negotiation successful
0929 16:47:19.876215 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":696,"thread_start_us":418,"threads_started":1}
I20260929 16:47:19.882110 789 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:47:19.882179 22842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7e28d8f88e942f2a5a1c9d841220787: Bootstrap starting.
I20260929 16:47:19.885663 22842 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7e28d8f88e942f2a5a1c9d841220787: Neither blocks nor log segments found. Creating new log.
I20260929 16:47:19.886998 22842 log.cc:824] T 00000000000000000000000000000000 P c7e28d8f88e942f2a5a1c9d841220787: Log is configured to *not* fsync() on all Append() calls
I20260929 16:47:19.890074 22845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:19.887076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40053 (local address 127.0.197.126:43457)
0929 16:47:19.887291 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:19.887297 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:19.887632 (+ 335us) server_negotiation.cc:407] Connection header received
0929 16:47:19.887919 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:19.887928 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:19.888043 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:19.888174 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:19.889262 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:47:19.889267 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:47:19.889270 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:47:19.889288 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:47:19.889330 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:47:19.889345 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:47:19.889444 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:19.889630 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 16:47:19.889688 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260929 16:47:19.891155 22842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7e28d8f88e942f2a5a1c9d841220787: No bootstrap required, opened a new log
I20260929 16:47:19.899493 22842 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7e28d8f88e942f2a5a1c9d841220787 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7e28d8f88e942f2a5a1c9d841220787" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 43457 } }
I20260929 16:47:19.899941 22842 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7e28d8f88e942f2a5a1c9d841220787 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:47:19.900060 22842 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7e28d8f88e942f2a5a1c9d841220787 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7e28d8f88e942f2a5a1c9d841220787, State: Initialized, Role: FOLLOWER
I20260929 16:47:19.900732 22842 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7e28d8f88e942f2a5a1c9d841220787 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7e28d8f88e942f2a5a1c9d841220787" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 43457 } }
I20260929 16:47:19.901010 22842 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7e28d8f88e942f2a5a1c9d841220787 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:47:19.901163 22842 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7e28d8f88e942f2a5a1c9d841220787 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:47:19.901340 22842 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7e28d8f88e942f2a5a1c9d841220787 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:47:19.903898 22842 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7e28d8f88e942f2a5a1c9d841220787 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7e28d8f88e942f2a5a1c9d841220787" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 43457 } }
I20260929 16:47:19.904449 22842 leader_election.cc:302] T 00000000000000000000000000000000 P c7e28d8f88e942f2a5a1c9d841220787 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7e28d8f88e942f2a5a1c9d841220787; no voters:
I20260929 16:47:19.905053 22842 leader_election.cc:288] T 00000000000000000000000000000000 P c7e28d8f88e942f2a5a1c9d841220787 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:47:19.905236 22852 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7e28d8f88e942f2a5a1c9d841220787 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:47:19.905975 22852 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7e28d8f88e942f2a5a1c9d841220787 [term 1 LEADER]: Becoming Leader. State: Replica: c7e28d8f88e942f2a5a1c9d841220787, State: Running, Role: LEADER
I20260929 16:47:19.906580 22852 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7e28d8f88e942f2a5a1c9d841220787 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7e28d8f88e942f2a5a1c9d841220787" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 43457 } }
I20260929 16:47:19.907285 22842 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7e28d8f88e942f2a5a1c9d841220787 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:47:19.909435 22854 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7e28d8f88e942f2a5a1c9d841220787 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7e28d8f88e942f2a5a1c9d841220787. Latest consensus state: current_term: 1 leader_uuid: "c7e28d8f88e942f2a5a1c9d841220787" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7e28d8f88e942f2a5a1c9d841220787" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 43457 } } }
I20260929 16:47:19.909916 22854 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7e28d8f88e942f2a5a1c9d841220787 [sys.catalog]: This master's current role is: LEADER
I20260929 16:47:19.915143 22853 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7e28d8f88e942f2a5a1c9d841220787 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7e28d8f88e942f2a5a1c9d841220787" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7e28d8f88e942f2a5a1c9d841220787" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 43457 } } }
I20260929 16:47:19.915531 22853 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7e28d8f88e942f2a5a1c9d841220787 [sys.catalog]: This master's current role is: LEADER
I20260929 16:47:19.916823 22859 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:47:19.922796 22859 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:47:19.933450 22859 catalog_manager.cc:1347] Generated new cluster ID: 4e63f52b73724b548198104886380e88
I20260929 16:47:19.933624 22859 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:47:19.954933 22859 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:47:19.955552 22859 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:47:19.967989 22859 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7e28d8f88e942f2a5a1c9d841220787: Generated new TSK 0
I20260929 16:47:19.969164 22859 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:47:19.984544 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 22775
W20260929 16:47:19.986320 22837 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46185
W20260929 16:47:19.987152 789 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 22775: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:47:19.987331 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 22775
2026-09-29T16:47:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2015 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790700068255611-789-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:47:20 dist-test-slave-l31f krb5kdc[22873](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:47:20 dist-test-slave-l31f krb5kdc[22873](info): set up 2 sockets
Sep 29 16:47:20 dist-test-slave-l31f krb5kdc[22873](info): commencing operation
krb5kdc: starting...
W20260929 16:47:21.119601 789 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790700068255611-789-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790700068255611-789-0/krb5kdc/test-user.keytab.
Sep 29 16:47:21 dist-test-slave-l31f krb5kdc[22873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:47:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:47:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.197.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
I20260929 16:47:21.411625 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:35531
--webserver_interface=127.0.197.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:45125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.126:35531
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:47:21.767263 22886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:47:21.767666 22886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:47:21.767836 22886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:47:21.775784 22886 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:47:21.775986 22886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:47:21.776095 22886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:47:21.776211 22886 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:47:21.776288 22886 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:47:21.787086 22886 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:45125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790700068255611-789-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.197.126:35531
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:35531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.197.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:47:21.788640 22886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:47:21.790059 22886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:47:21.804062 22891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:47:21.803941 22892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:47:21.804032 22894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:47:21.805099 22886 server_base.cc:1034] running on GCE node
I20260929 16:47:21.805939 22886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:47:21.807650 22886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:47:21.808878 22886 hybrid_clock.cc:648] HybridClock initialized: now 1790700441808835 us; error 35 us; skew 500 ppm
I20260929 16:47:21.809249 22886 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:47:21 dist-test-slave-l31f krb5kdc[22873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:47:21.812721 22886 init.cc:377] Logged in from keytab as kudu/127.0.197.126@KRBTEST.COM (short username kudu)
I20260929 16:47:21.814455 22886 webserver.cc:460] Webserver started at http://127.0.197.126:39777/ using document root <none> and password file <none>
I20260929 16:47:21.815307 22886 fs_manager.cc:359] Metadata directory not provided
I20260929 16:47:21.815488 22886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:47:21.815857 22886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:47:21.819271 22886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790700068255611-789-0/minicluster-data/master-0/data/instance:
uuid: "17c5a7bd7ed240a1b911e6151cdd6fab"
format_stamp: "Formatted at 2026-09-29 16:47:21 on dist-test-slave-l31f"
server_key: "a774a8cfa87ae0f7536381d29c5de40e"
server_key_iv: "3b17f7cc6ef969e0514da0d89bf25d1a"
server_key_version: "encryptionkey@0"
I20260929 16:47:21.820189 22886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790700068255611-789-0/minicluster-data/master-0/wal/instance:
uuid: "17c5a7bd7ed240a1b911e6151cdd6fab"
format_stamp: "Formatted at 2026-09-29 16:47:21 on dist-test-slave-l31f"
server_key: "a774a8cfa87ae0f7536381d29c5de40e"
server_key_iv: "3b17f7cc6ef969e0514da0d89bf25d1a"
server_key_version: "encryptionkey@0"
I20260929 16:47:21.826316 22886 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260929 16:47:21.830811 22901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:47:21.832578 22886 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:47:21.832828 22886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "17c5a7bd7ed240a1b911e6151cdd6fab"
format_stamp: "Formatted at 2026-09-29 16:47:21 on dist-test-slave-l31f"
server_key: "a774a8cfa87ae0f7536381d29c5de40e"
server_key_iv: "3b17f7cc6ef969e0514da0d89bf25d1a"
server_key_version: "encryptionkey@0"
I20260929 16:47:21.833086 22886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:47:21.853991 22886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:47:21.859160 22886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:47:21.859503 22886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:47:21.880862 22886 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.126:35531
I20260929 16:47:21.880882 22952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.126:35531 every 8 connection(s)
I20260929 16:47:21.882400 22886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:47:21.886593 22953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:47:21.890944 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 22886
I20260929 16:47:21.891167 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790700068255611-789-0/minicluster-data/master-0/wal/instance
I20260929 16:47:21.891721 789 external_mini_cluster.cc:1428] Setting key 8d5e82e58250cadd7949abf8b677ce24
I20260929 16:47:21.901361 22953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17c5a7bd7ed240a1b911e6151cdd6fab: Bootstrap starting.
I20260929 16:47:21.905014 22953 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17c5a7bd7ed240a1b911e6151cdd6fab: Neither blocks nor log segments found. Creating new log.
I20260929 16:47:21.906243 22953 log.cc:824] T 00000000000000000000000000000000 P 17c5a7bd7ed240a1b911e6151cdd6fab: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:47:21 dist-test-slave-l31f krb5kdc[22873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:47:21.909469 22953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17c5a7bd7ed240a1b911e6151cdd6fab: No bootstrap required, opened a new log
I20260929 16:47:21.918473 22956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:21.893903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52809 (local address 127.0.197.126:35531)
0929 16:47:21.894932 (+ 1029us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:21.894946 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:21.894997 (+ 51us) server_negotiation.cc:407] Connection header received
0929 16:47:21.897043 (+ 2046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:21.897073 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:21.897603 (+ 530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:21.898123 (+ 520us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:21.900178 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:21.901829 (+ 1651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:21.903519 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:21.904336 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:21.904580 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:21.909576 (+ 4996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:47:21.909614 (+ 38us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:47:21.909639 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:47:21.909702 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:47:21.913550 (+ 3848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:21.914480 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:21.914488 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:21.914495 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:21.914707 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:21.915216 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:21.915220 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:21.915222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:21.915981 (+ 759us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:47:21.916359 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:21.916866 (+ 507us) server_negotiation.cc:299] Negotiation successful
0929 16:47:21.917356 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":782,"thread_start_us":535,"threads_started":1}
I20260929 16:47:21.918612 22953 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17c5a7bd7ed240a1b911e6151cdd6fab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c5a7bd7ed240a1b911e6151cdd6fab" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 35531 } }
I20260929 16:47:21.919212 22953 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17c5a7bd7ed240a1b911e6151cdd6fab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:47:21.919341 22953 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17c5a7bd7ed240a1b911e6151cdd6fab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17c5a7bd7ed240a1b911e6151cdd6fab, State: Initialized, Role: FOLLOWER
I20260929 16:47:21.920065 22953 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17c5a7bd7ed240a1b911e6151cdd6fab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c5a7bd7ed240a1b911e6151cdd6fab" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 35531 } }
I20260929 16:47:21.920356 22953 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17c5a7bd7ed240a1b911e6151cdd6fab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:47:21.920526 22953 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17c5a7bd7ed240a1b911e6151cdd6fab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:47:21.920730 22953 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17c5a7bd7ed240a1b911e6151cdd6fab [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:47:21.923053 789 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:47:21.923667 22953 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17c5a7bd7ed240a1b911e6151cdd6fab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c5a7bd7ed240a1b911e6151cdd6fab" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 35531 } }
I20260929 16:47:21.924250 22953 leader_election.cc:302] T 00000000000000000000000000000000 P 17c5a7bd7ed240a1b911e6151cdd6fab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17c5a7bd7ed240a1b911e6151cdd6fab; no voters:
I20260929 16:47:21.924889 22953 leader_election.cc:288] T 00000000000000000000000000000000 P 17c5a7bd7ed240a1b911e6151cdd6fab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:47:21.925114 22958 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17c5a7bd7ed240a1b911e6151cdd6fab [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:47:21.926388 22958 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17c5a7bd7ed240a1b911e6151cdd6fab [term 1 LEADER]: Becoming Leader. State: Replica: 17c5a7bd7ed240a1b911e6151cdd6fab, State: Running, Role: LEADER
I20260929 16:47:21.927386 22953 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17c5a7bd7ed240a1b911e6151cdd6fab [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:47:21.927284 22958 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17c5a7bd7ed240a1b911e6151cdd6fab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c5a7bd7ed240a1b911e6151cdd6fab" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 35531 } }
I20260929 16:47:21.931187 22960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17c5a7bd7ed240a1b911e6151cdd6fab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17c5a7bd7ed240a1b911e6151cdd6fab. Latest consensus state: current_term: 1 leader_uuid: "17c5a7bd7ed240a1b911e6151cdd6fab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c5a7bd7ed240a1b911e6151cdd6fab" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 35531 } } }
I20260929 16:47:21.931797 22960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17c5a7bd7ed240a1b911e6151cdd6fab [sys.catalog]: This master's current role is: LEADER
I20260929 16:47:21.932495 22959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17c5a7bd7ed240a1b911e6151cdd6fab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17c5a7bd7ed240a1b911e6151cdd6fab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17c5a7bd7ed240a1b911e6151cdd6fab" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 35531 } } }
I20260929 16:47:21.932850 22959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17c5a7bd7ed240a1b911e6151cdd6fab [sys.catalog]: This master's current role is: LEADER
I20260929 16:47:21.939779 22968 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:47:21.953016 22956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:21.933659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38741 (local address 127.0.197.126:35531)
0929 16:47:21.933908 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:21.933918 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:21.934261 (+ 343us) server_negotiation.cc:407] Connection header received
0929 16:47:21.934486 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:21.934492 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:21.934575 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:21.934688 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:21.936604 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:21.937530 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:21.939367 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:21.940120 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:21.940232 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:21.944453 (+ 4221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:47:21.944465 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:47:21.944473 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:47:21.944541 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:47:21.949094 (+ 4553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:21.950184 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:21.950193 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:21.950197 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:21.950408 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:21.950949 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:21.950955 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:21.950960 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:21.951464 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:47:21.951720 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:21.952347 (+ 627us) server_negotiation.cc:299] Negotiation successful
0929 16:47:21.952622 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":80}
I20260929 16:47:21.955252 22968 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:47:21.967674 22968 catalog_manager.cc:1347] Generated new cluster ID: e82d0a5a963649bea30abc44de6bcc2b
I20260929 16:47:21.967837 22968 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:47:21.993005 22968 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:47:21.994455 22968 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:47:22.005007 22968 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17c5a7bd7ed240a1b911e6151cdd6fab: Generated new TSK 0
I20260929 16:47:22.006278 22968 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:47:22.019842 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 22886
I20260929 16:47:22.021376 22948 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:47:22.122264 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 22886
2026-09-29T16:47:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2137 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790700068255611-789-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:47:22 dist-test-slave-l31f krb5kdc[22985](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:47:22 dist-test-slave-l31f krb5kdc[22985](info): set up 2 sockets
Sep 29 16:47:22 dist-test-slave-l31f krb5kdc[22985](info): commencing operation
krb5kdc: starting...
W20260929 16:47:23.248202 789 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.002s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790700068255611-789-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790700068255611-789-0/krb5kdc/test-user.keytab.
Sep 29 16:47:23 dist-test-slave-l31f krb5kdc[22985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:47:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:47:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.197.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
I20260929 16:47:23.528604 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:38535
--webserver_interface=127.0.197.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:38069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.126:38535
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:47:23.887869 22998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:47:23.888255 22998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:47:23.888353 22998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:47:23.896108 22998 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:47:23.896248 22998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:47:23.896296 22998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:47:23.896350 22998 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:47:23.896394 22998 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:47:23.906965 22998 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:38069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790700068255611-789-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.197.126:38535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:38535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.197.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:47:23.908488 22998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:47:23.910185 22998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:47:23.923030 23003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:47:23.923053 23004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:47:23.922967 23006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:47:23.923910 22998 server_base.cc:1034] running on GCE node
I20260929 16:47:23.924834 22998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:47:23.926419 22998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:47:23.927659 22998 hybrid_clock.cc:648] HybridClock initialized: now 1790700443927592 us; error 60 us; skew 500 ppm
I20260929 16:47:23.928074 22998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:47:23 dist-test-slave-l31f krb5kdc[22985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:47:23.937695 22998 init.cc:377] Logged in from keytab as kudu/127.0.197.126@KRBTEST.COM (short username kudu)
I20260929 16:47:23.939740 22998 webserver.cc:460] Webserver started at http://127.0.197.126:42121/ using document root <none> and password file <none>
I20260929 16:47:23.940529 22998 fs_manager.cc:359] Metadata directory not provided
I20260929 16:47:23.940689 22998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:47:23.941043 22998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:47:23.944626 22998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790700068255611-789-0/minicluster-data/master-0/data/instance:
uuid: "0c2435596ecf44748944af0f1c1902fb"
format_stamp: "Formatted at 2026-09-29 16:47:23 on dist-test-slave-l31f"
server_key: "6353ef2c098ceb91dcd8e0c4088c0c9b"
server_key_iv: "20fe6ef1a3f276f649ead70c8fdf8899"
server_key_version: "encryptionkey@0"
I20260929 16:47:23.945560 22998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790700068255611-789-0/minicluster-data/master-0/wal/instance:
uuid: "0c2435596ecf44748944af0f1c1902fb"
format_stamp: "Formatted at 2026-09-29 16:47:23 on dist-test-slave-l31f"
server_key: "6353ef2c098ceb91dcd8e0c4088c0c9b"
server_key_iv: "20fe6ef1a3f276f649ead70c8fdf8899"
server_key_version: "encryptionkey@0"
I20260929 16:47:23.951540 22998 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260929 16:47:23.955511 23013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:47:23.957105 22998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:47:23.957334 22998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "0c2435596ecf44748944af0f1c1902fb"
format_stamp: "Formatted at 2026-09-29 16:47:23 on dist-test-slave-l31f"
server_key: "6353ef2c098ceb91dcd8e0c4088c0c9b"
server_key_iv: "20fe6ef1a3f276f649ead70c8fdf8899"
server_key_version: "encryptionkey@0"
I20260929 16:47:23.957568 22998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:47:23.974412 22998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:47:23.979275 22998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:47:23.979614 22998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:47:24.000044 22998 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.126:38535
I20260929 16:47:24.000059 23064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.126:38535 every 8 connection(s)
I20260929 16:47:24.001570 22998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:47:24.005738 23065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:47:24.007107 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 22998
I20260929 16:47:24.007304 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790700068255611-789-0/minicluster-data/master-0/wal/instance
I20260929 16:47:24.007851 789 external_mini_cluster.cc:1428] Setting key 4979c50623a6c1bbf6f2caee22a626b1
I20260929 16:47:24.019419 23065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c2435596ecf44748944af0f1c1902fb: Bootstrap starting.
I20260929 16:47:24.021739 23068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:24.009938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48673 (local address 127.0.197.126:38535)
0929 16:47:24.010795 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:24.010839 (+ 44us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:24.010895 (+ 56us) server_negotiation.cc:407] Connection header received
0929 16:47:24.012723 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:24.012754 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:24.013210 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:24.013649 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:24.014231 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:24.015592 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:24.016919 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:24.017631 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:24.017857 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:24.019205 (+ 1348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:47:24.019213 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:47:24.019229 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:47:24.019856 (+ 627us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:47:24.019960 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:47:24.019988 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:47:24.020123 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:24.020359 (+ 236us) server_negotiation.cc:299] Negotiation successful
0929 16:47:24.020648 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":362,"threads_started":1}
I20260929 16:47:24.023052 23065 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c2435596ecf44748944af0f1c1902fb: Neither blocks nor log segments found. Creating new log.
I20260929 16:47:24.024271 23065 log.cc:824] T 00000000000000000000000000000000 P 0c2435596ecf44748944af0f1c1902fb: Log is configured to *not* fsync() on all Append() calls
I20260929 16:47:24.026345 789 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:47:24.027983 23065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c2435596ecf44748944af0f1c1902fb: No bootstrap required, opened a new log
I20260929 16:47:24.038543 23068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:24.031248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47367 (local address 127.0.197.126:38535)
0929 16:47:24.031520 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:24.031530 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:24.031986 (+ 456us) server_negotiation.cc:407] Connection header received
0929 16:47:24.032193 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:24.032197 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:24.032268 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:24.032363 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:24.032992 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:24.033828 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:24.035876 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:24.036355 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:24.036488 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:24.037721 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:47:24.037726 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:47:24.037729 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:47:24.037749 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:47:24.037791 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:47:24.037807 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:47:24.037879 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:24.038077 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 16:47:24.038160 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 16:47:24.038203 23065 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c2435596ecf44748944af0f1c1902fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c2435596ecf44748944af0f1c1902fb" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 38535 } }
I20260929 16:47:24.038745 23065 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c2435596ecf44748944af0f1c1902fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:47:24.038934 23065 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c2435596ecf44748944af0f1c1902fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c2435596ecf44748944af0f1c1902fb, State: Initialized, Role: FOLLOWER
I20260929 16:47:24.039685 23065 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c2435596ecf44748944af0f1c1902fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c2435596ecf44748944af0f1c1902fb" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 38535 } }
I20260929 16:47:24.040004 23065 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c2435596ecf44748944af0f1c1902fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:47:24.040159 23065 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c2435596ecf44748944af0f1c1902fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:47:24.040380 23065 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c2435596ecf44748944af0f1c1902fb [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:47:24.042750 23065 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c2435596ecf44748944af0f1c1902fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c2435596ecf44748944af0f1c1902fb" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 38535 } }
I20260929 16:47:24.043421 23065 leader_election.cc:302] T 00000000000000000000000000000000 P 0c2435596ecf44748944af0f1c1902fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c2435596ecf44748944af0f1c1902fb; no voters:
I20260929 16:47:24.044059 23065 leader_election.cc:288] T 00000000000000000000000000000000 P 0c2435596ecf44748944af0f1c1902fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:47:24.044194 23075 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c2435596ecf44748944af0f1c1902fb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:47:24.045629 23075 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c2435596ecf44748944af0f1c1902fb [term 1 LEADER]: Becoming Leader. State: Replica: 0c2435596ecf44748944af0f1c1902fb, State: Running, Role: LEADER
I20260929 16:47:24.046327 23065 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c2435596ecf44748944af0f1c1902fb [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:47:24.046299 23075 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c2435596ecf44748944af0f1c1902fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c2435596ecf44748944af0f1c1902fb" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 38535 } }
I20260929 16:47:24.049415 23077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c2435596ecf44748944af0f1c1902fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c2435596ecf44748944af0f1c1902fb. Latest consensus state: current_term: 1 leader_uuid: "0c2435596ecf44748944af0f1c1902fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c2435596ecf44748944af0f1c1902fb" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 38535 } } }
I20260929 16:47:24.049868 23077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c2435596ecf44748944af0f1c1902fb [sys.catalog]: This master's current role is: LEADER
I20260929 16:47:24.050393 23076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c2435596ecf44748944af0f1c1902fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c2435596ecf44748944af0f1c1902fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c2435596ecf44748944af0f1c1902fb" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 38535 } } }
I20260929 16:47:24.050675 23076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c2435596ecf44748944af0f1c1902fb [sys.catalog]: This master's current role is: LEADER
I20260929 16:47:24.052668 23084 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:47:24.057888 23084 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:47:24.067477 23084 catalog_manager.cc:1347] Generated new cluster ID: c36eb0cb4a62452896370537c07f723c
I20260929 16:47:24.067639 23084 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:47:24.082162 23084 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:47:24.084262 23084 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:47:24.094527 23084 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c2435596ecf44748944af0f1c1902fb: Generated new TSK 0
I20260929 16:47:24.095613 23084 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:47:24.107360 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 22998
W20260929 16:47:24.108611 23060 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48673
W20260929 16:47:24.109452 789 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 22998: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:47:24.109634 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 22998
2026-09-29T16:47:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1987 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790700068255611-789-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:47:24 dist-test-slave-l31f krb5kdc[23096](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:47:24 dist-test-slave-l31f krb5kdc[23096](info): set up 2 sockets
Sep 29 16:47:24 dist-test-slave-l31f krb5kdc[23096](info): commencing operation
krb5kdc: starting...
W20260929 16:47:25.243765 789 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.011s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790700068255611-789-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790700068255611-789-0/krb5kdc/test-user.keytab.
Sep 29 16:47:25 dist-test-slave-l31f krb5kdc[23096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:47:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:47:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.197.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
I20260929 16:47:25.531617 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:39063
--webserver_interface=127.0.197.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:45013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.126:39063
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:47:25.886669 23109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:47:25.886999 23109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:47:25.887248 23109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:47:25.895239 23109 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:47:25.895383 23109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:47:25.895483 23109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:47:25.895586 23109 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:47:25.895663 23109 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:47:25.906242 23109 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:45013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790700068255611-789-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.197.126:39063
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:39063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.197.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:47:25.907711 23109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:47:25.909099 23109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:47:25.921594 23115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:47:25.921563 23114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:47:25.921782 23117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:47:25.922675 23109 server_base.cc:1034] running on GCE node
I20260929 16:47:25.923689 23109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:47:25.925208 23109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:47:25.926504 23109 hybrid_clock.cc:648] HybridClock initialized: now 1790700445926428 us; error 42 us; skew 500 ppm
I20260929 16:47:25.926884 23109 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:47:25 dist-test-slave-l31f krb5kdc[23096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:47:25.930059 23109 init.cc:377] Logged in from keytab as kudu/127.0.197.126@KRBTEST.COM (short username kudu)
I20260929 16:47:25.931721 23109 webserver.cc:460] Webserver started at http://127.0.197.126:40837/ using document root <none> and password file <none>
I20260929 16:47:25.932468 23109 fs_manager.cc:359] Metadata directory not provided
I20260929 16:47:25.932632 23109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:47:25.932955 23109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:47:25.936170 23109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790700068255611-789-0/minicluster-data/master-0/data/instance:
uuid: "d00da4fabd8d4fb5858b7461ef043a59"
format_stamp: "Formatted at 2026-09-29 16:47:25 on dist-test-slave-l31f"
server_key: "0e40e542601e5203503a9b3d25acf094"
server_key_iv: "032968c7952d919e63f28632836b9556"
server_key_version: "encryptionkey@0"
I20260929 16:47:25.936993 23109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790700068255611-789-0/minicluster-data/master-0/wal/instance:
uuid: "d00da4fabd8d4fb5858b7461ef043a59"
format_stamp: "Formatted at 2026-09-29 16:47:25 on dist-test-slave-l31f"
server_key: "0e40e542601e5203503a9b3d25acf094"
server_key_iv: "032968c7952d919e63f28632836b9556"
server_key_version: "encryptionkey@0"
I20260929 16:47:25.942525 23109 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260929 16:47:25.946393 23124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:47:25.947872 23109 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:47:25.948081 23109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "d00da4fabd8d4fb5858b7461ef043a59"
format_stamp: "Formatted at 2026-09-29 16:47:25 on dist-test-slave-l31f"
server_key: "0e40e542601e5203503a9b3d25acf094"
server_key_iv: "032968c7952d919e63f28632836b9556"
server_key_version: "encryptionkey@0"
I20260929 16:47:25.948303 23109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:47:25.954290 23109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:47:25.958454 23109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:47:25.958776 23109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:47:25.978082 23109 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.126:39063
I20260929 16:47:25.978113 23175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.126:39063 every 8 connection(s)
I20260929 16:47:25.979581 23109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:47:25.980592 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 23109
I20260929 16:47:25.980782 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790700068255611-789-0/minicluster-data/master-0/wal/instance
I20260929 16:47:25.981351 789 external_mini_cluster.cc:1428] Setting key 246acf684a3478297a10b1170f86dabe
I20260929 16:47:25.984368 23176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:47:25.989152 23179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:25.983155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52049 (local address 127.0.197.126:39063)
0929 16:47:25.983901 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:25.983913 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:25.983953 (+ 40us) server_negotiation.cc:407] Connection header received
0929 16:47:25.985621 (+ 1668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:25.985644 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:25.985979 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:25.986357 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:25.987484 (+ 1127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:47:25.987490 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:47:25.987503 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:47:25.987523 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:47:25.987589 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:47:25.987610 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:47:25.987885 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:25.988025 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 16:47:25.988205 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":325,"threads_started":1}
I20260929 16:47:25.993283 789 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:47:25.996670 23176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d00da4fabd8d4fb5858b7461ef043a59: Bootstrap starting.
I20260929 16:47:26.000416 23176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d00da4fabd8d4fb5858b7461ef043a59: Neither blocks nor log segments found. Creating new log.
I20260929 16:47:26.000661 23179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:25.997821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57293 (local address 127.0.197.126:39063)
0929 16:47:25.998104 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:25.998112 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:25.998448 (+ 336us) server_negotiation.cc:407] Connection header received
0929 16:47:25.998655 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:25.998664 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:25.998715 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:25.998833 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:25.999840 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:47:25.999846 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:47:25.999851 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:47:25.999877 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:47:25.999921 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:47:25.999940 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:47:26.000072 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:26.000179 (+ 107us) server_negotiation.cc:299] Negotiation successful
0929 16:47:26.000244 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
I20260929 16:47:26.001902 23176 log.cc:824] T 00000000000000000000000000000000 P d00da4fabd8d4fb5858b7461ef043a59: Log is configured to *not* fsync() on all Append() calls
I20260929 16:47:26.005291 23176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d00da4fabd8d4fb5858b7461ef043a59: No bootstrap required, opened a new log
I20260929 16:47:26.014663 23176 raft_consensus.cc:348] T 00000000000000000000000000000000 P d00da4fabd8d4fb5858b7461ef043a59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d00da4fabd8d4fb5858b7461ef043a59" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 39063 } }
I20260929 16:47:26.015218 23176 raft_consensus.cc:374] T 00000000000000000000000000000000 P d00da4fabd8d4fb5858b7461ef043a59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:47:26.015336 23176 raft_consensus.cc:729] T 00000000000000000000000000000000 P d00da4fabd8d4fb5858b7461ef043a59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d00da4fabd8d4fb5858b7461ef043a59, State: Initialized, Role: FOLLOWER
I20260929 16:47:26.015992 23176 consensus_queue.cc:260] T 00000000000000000000000000000000 P d00da4fabd8d4fb5858b7461ef043a59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d00da4fabd8d4fb5858b7461ef043a59" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 39063 } }
I20260929 16:47:26.016263 23176 raft_consensus.cc:388] T 00000000000000000000000000000000 P d00da4fabd8d4fb5858b7461ef043a59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:47:26.016439 23176 raft_consensus.cc:482] T 00000000000000000000000000000000 P d00da4fabd8d4fb5858b7461ef043a59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:47:26.016629 23176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d00da4fabd8d4fb5858b7461ef043a59 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:47:26.019063 23176 raft_consensus.cc:504] T 00000000000000000000000000000000 P d00da4fabd8d4fb5858b7461ef043a59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d00da4fabd8d4fb5858b7461ef043a59" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 39063 } }
I20260929 16:47:26.019666 23176 leader_election.cc:302] T 00000000000000000000000000000000 P d00da4fabd8d4fb5858b7461ef043a59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d00da4fabd8d4fb5858b7461ef043a59; no voters:
I20260929 16:47:26.020413 23176 leader_election.cc:288] T 00000000000000000000000000000000 P d00da4fabd8d4fb5858b7461ef043a59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:47:26.020579 23186 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d00da4fabd8d4fb5858b7461ef043a59 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:47:26.022027 23186 raft_consensus.cc:686] T 00000000000000000000000000000000 P d00da4fabd8d4fb5858b7461ef043a59 [term 1 LEADER]: Becoming Leader. State: Replica: d00da4fabd8d4fb5858b7461ef043a59, State: Running, Role: LEADER
I20260929 16:47:26.022596 23186 consensus_queue.cc:237] T 00000000000000000000000000000000 P d00da4fabd8d4fb5858b7461ef043a59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d00da4fabd8d4fb5858b7461ef043a59" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 39063 } }
I20260929 16:47:26.022899 23176 sys_catalog.cc:564] T 00000000000000000000000000000000 P d00da4fabd8d4fb5858b7461ef043a59 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:47:26.026154 23187 sys_catalog.cc:455] T 00000000000000000000000000000000 P d00da4fabd8d4fb5858b7461ef043a59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d00da4fabd8d4fb5858b7461ef043a59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d00da4fabd8d4fb5858b7461ef043a59" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 39063 } } }
I20260929 16:47:26.026646 23187 sys_catalog.cc:458] T 00000000000000000000000000000000 P d00da4fabd8d4fb5858b7461ef043a59 [sys.catalog]: This master's current role is: LEADER
I20260929 16:47:26.027261 23188 sys_catalog.cc:455] T 00000000000000000000000000000000 P d00da4fabd8d4fb5858b7461ef043a59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d00da4fabd8d4fb5858b7461ef043a59. Latest consensus state: current_term: 1 leader_uuid: "d00da4fabd8d4fb5858b7461ef043a59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d00da4fabd8d4fb5858b7461ef043a59" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 39063 } } }
I20260929 16:47:26.027562 23195 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:47:26.027623 23188 sys_catalog.cc:458] T 00000000000000000000000000000000 P d00da4fabd8d4fb5858b7461ef043a59 [sys.catalog]: This master's current role is: LEADER
I20260929 16:47:26.037065 23195 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:47:26.046809 23195 catalog_manager.cc:1347] Generated new cluster ID: 5dc45e1f69b74e09aabd1d1cd0aa17b1
I20260929 16:47:26.046963 23195 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:47:26.067518 23195 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:47:26.068178 23195 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:47:26.077401 23195 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d00da4fabd8d4fb5858b7461ef043a59: Generated new TSK 0
I20260929 16:47:26.078631 23195 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:47:26.087441 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 23109
W20260929 16:47:26.089311 23171 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52049
W20260929 16:47:26.090199 789 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 23109: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:47:26.090379 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 23109
2026-09-29T16:47:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1981 ms)
[----------] 18 tests from AuthTokenIssuingTest (36957 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:47:26 dist-test-slave-l31f krb5kdc[23207](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:47:26 dist-test-slave-l31f krb5kdc[23207](info): set up 2 sockets
Sep 29 16:47:26 dist-test-slave-l31f krb5kdc[23207](info): commencing operation
krb5kdc: starting...
W20260929 16:47:27.223739 789 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/krb5kdc/test-user.keytab.
Sep 29 16:47:27 dist-test-slave-l31f krb5kdc[23207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:47:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:47:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.197.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
I20260929 16:47:27.520188 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:33615
--webserver_interface=127.0.197.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:43335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.126:33615,127.0.197.125:41099,127.0.197.124:43025
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:47:27.878333 23220 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 16:47:27.878818 23220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:47:27.878949 23220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:47:27.879137 23220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:47:27.887079 23220 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:47:27.887249 23220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:47:27.887359 23220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:47:27.887461 23220 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:47:27.887557 23220 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:47:27.898044 23220 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:43335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.197.126:33615,127.0.197.125:41099,127.0.197.124:43025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:33615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.197.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:47:27.899566 23220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:47:27.900985 23220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:47:27.915164 23225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:47:27.915287 23228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:47:27.915323 23226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:47:27.915829 23220 server_base.cc:1034] running on GCE node
I20260929 16:47:27.916749 23220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:47:27.918332 23220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:47:27.919570 23220 hybrid_clock.cc:648] HybridClock initialized: now 1790700447919512 us; error 34 us; skew 500 ppm
I20260929 16:47:27.920023 23220 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:47:27 dist-test-slave-l31f krb5kdc[23207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:47:27.923439 23220 init.cc:377] Logged in from keytab as kudu/127.0.197.126@KRBTEST.COM (short username kudu)
I20260929 16:47:27.925145 23220 webserver.cc:460] Webserver started at http://127.0.197.126:46349/ using document root <none> and password file <none>
I20260929 16:47:27.925918 23220 fs_manager.cc:359] Metadata directory not provided
I20260929 16:47:27.926075 23220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:47:27.926415 23220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:47:27.929781 23220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/minicluster-data/master-0/data/instance:
uuid: "57843dd206c24c658e60f97672cfc60c"
format_stamp: "Formatted at 2026-09-29 16:47:27 on dist-test-slave-l31f"
server_key: "a0f006a821a5f4cf5ed05cd2999e6c61"
server_key_iv: "cbe27b2ed59830b5deee29a59bee6276"
server_key_version: "encryptionkey@0"
I20260929 16:47:27.930652 23220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/minicluster-data/master-0/wal/instance:
uuid: "57843dd206c24c658e60f97672cfc60c"
format_stamp: "Formatted at 2026-09-29 16:47:27 on dist-test-slave-l31f"
server_key: "a0f006a821a5f4cf5ed05cd2999e6c61"
server_key_iv: "cbe27b2ed59830b5deee29a59bee6276"
server_key_version: "encryptionkey@0"
I20260929 16:47:27.936619 23220 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 16:47:27.940490 23235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:47:27.942042 23220 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:47:27.942268 23220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "57843dd206c24c658e60f97672cfc60c"
format_stamp: "Formatted at 2026-09-29 16:47:27 on dist-test-slave-l31f"
server_key: "a0f006a821a5f4cf5ed05cd2999e6c61"
server_key_iv: "cbe27b2ed59830b5deee29a59bee6276"
server_key_version: "encryptionkey@0"
I20260929 16:47:27.942536 23220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:47:27.961812 23220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:47:27.966434 23220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:47:27.966774 23220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:47:27.987113 23220 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.126:33615
I20260929 16:47:27.987177 23286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.126:33615 every 8 connection(s)
I20260929 16:47:27.988612 23220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:47:27.992642 23287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:47:27.993359 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 23220
I20260929 16:47:27.993602 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/minicluster-data/master-0/wal/instance
I20260929 16:47:27.994290 789 external_mini_cluster.cc:1428] Setting key 8ada2c820b8fdee574fa76f8b3b4464b
I20260929 16:47:28.000419 23287 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33615 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:47:28 dist-test-slave-l31f krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.197.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.125@KRBTEST.COM" created.
I20260929 16:47:28.028471 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:28.003943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51784 (local address 127.0.197.126:33615)
0929 16:47:28.005832 (+ 1889us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:28.005841 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:28.005901 (+ 60us) server_negotiation.cc:407] Connection header received
0929 16:47:28.007819 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:28.007832 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:28.008240 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:28.008493 (+ 253us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:28.009903 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:28.012071 (+ 2168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:28.013826 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:28.014919 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:28.015228 (+ 309us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:28.019471 (+ 4243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:47:28.019488 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:47:28.019510 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:47:28.019571 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:47:28.023339 (+ 3768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:28.024308 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:28.024316 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:28.024318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:28.024610 (+ 292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:28.025080 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:28.025084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:28.025086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:28.025646 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:47:28.025848 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:28.026382 (+ 534us) server_negotiation.cc:299] Negotiation successful
0929 16:47:28.026839 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":101,"server-negotiator.queue_time_us":1620,"thread_start_us":1396,"threads_started":1}
I20260929 16:47:28.028494 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:28.003942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:33615 (local address 127.0.0.1:51784)
0929 16:47:28.004908 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:28.004961 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 16:47:28.007239 (+ 2278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:47:28.008539 (+ 1300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:47:28.008545 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:47:28.009211 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:47:28.009624 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:47:28.009645 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:47:28.012335 (+ 2690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:47:28.012340 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:47:28.013610 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:47:28.013629 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:47:28.015311 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:47:28.015317 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:47:28.015504 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:28.016292 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:47:28.016319 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:47:28.019154 (+ 2835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:47:28.023634 (+ 4480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:47:28.023646 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:47:28.023657 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:47:28.024134 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:47:28.024749 (+ 615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:47:28.024753 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:47:28.024755 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:47:28.024919 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:47:28.025796 (+ 877us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:47:28.025804 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:47:28.026019 (+ 215us) client_negotiation.cc:769] Sending connection context
0929 16:47:28.026282 (+ 263us) client_negotiation.cc:240] Negotiation successful
0929 16:47:28.026967 (+ 685us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"mutex_wait_us":162,"thread_start_us":427,"threads_started":1}
I20260929 16:47:28.032320 23287 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.197.125" port: 41099 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:47:28.033782 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:28.033259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.125:41099 (local address 127.0.0.1:44748)
0929 16:47:28.033479 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:28.033599 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.125:41099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260929 16:47:28.034138 23237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.197.125:41099: connect: Connection refused (error 111)
I20260929 16:47:28.036026 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:28.035613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.125:41099 (local address 127.0.0.1:44762)
0929 16:47:28.035775 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:28.035872 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.125:41099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260929 16:47:28.036371 23287 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.197.125:41099: Network error: Client connection negotiation failed: client connection to 127.0.197.125:41099: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/krb5kdc/kudu_127.0.197.125.keytab.
Entry for principal kudu/127.0.197.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/krb5kdc/kudu_127.0.197.125.keytab.
I20260929 16:47:28.058791 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.125:41099
--webserver_interface=127.0.197.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/krb5kdc/kudu_127.0.197.125.keytab
--principal=kudu/127.0.197.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:43335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.126:33615,127.0.197.125:41099,127.0.197.124:43025
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:47:28.085794 23287 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.197.125" port: 41099 } attempt: 1
I20260929 16:47:28.087157 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:28.086621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.125:41099 (local address 127.0.0.1:44764)
0929 16:47:28.086826 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:28.086968 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.125:41099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260929 16:47:28.088361 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:28.087915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.125:41099 (local address 127.0.0.1:44780)
0929 16:47:28.088088 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:28.088183 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.125:41099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 16:47:28.088652 23287 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.197.125:41099: Network error: Client connection negotiation failed: client connection to 127.0.197.125:41099: connect: Connection refused (error 111)
I20260929 16:47:28.156996 23287 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.197.125" port: 41099 } attempt: 2
I20260929 16:47:28.158435 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:28.157867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.125:41099 (local address 127.0.0.1:44796)
0929 16:47:28.158168 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:28.158291 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.125:41099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260929 16:47:28.159655 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:28.159245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.125:41099 (local address 127.0.0.1:44804)
0929 16:47:28.159403 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:28.159506 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.125:41099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 16:47:28.160049 23287 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.197.125:41099: Network error: Client connection negotiation failed: client connection to 127.0.197.125:41099: connect: Connection refused (error 111)
I20260929 16:47:28.251370 23287 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.197.125" port: 41099 } attempt: 3
I20260929 16:47:28.252604 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:28.252119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.125:41099 (local address 127.0.0.1:44818)
0929 16:47:28.252369 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:28.252462 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.125:41099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260929 16:47:28.253714 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:28.253312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.125:41099 (local address 127.0.0.1:44830)
0929 16:47:28.253483 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:28.253566 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.125:41099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260929 16:47:28.254029 23287 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.197.125:41099: Network error: Client connection negotiation failed: client connection to 127.0.197.125:41099: connect: Connection refused (error 111)
I20260929 16:47:28.397370 23287 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.197.125" port: 41099 } attempt: 4
I20260929 16:47:28.398717 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:28.398121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.125:41099 (local address 127.0.0.1:44832)
0929 16:47:28.398472 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:28.398565 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.125:41099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200}
I20260929 16:47:28.399914 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:28.399521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.125:41099 (local address 127.0.0.1:44848)
0929 16:47:28.399709 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:28.399782 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.125:41099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260929 16:47:28.400171 23287 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.197.125:41099: Network error: Client connection negotiation failed: client connection to 127.0.197.125:41099: connect: Connection refused (error 111)
W20260929 16:47:28.414278 23294 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 16:47:28.414732 23294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:47:28.414815 23294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:47:28.414901 23294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:47:28.423132 23294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:47:28.423244 23294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:47:28.423303 23294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:47:28.423355 23294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:47:28.423420 23294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:47:28.434015 23294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:43335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.197.126:33615,127.0.197.125:41099,127.0.197.124:43025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/krb5kdc/kudu_127.0.197.125.keytab
--principal=kudu/127.0.197.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.125:41099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.197.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:47:28.435523 23294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:47:28.436863 23294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:47:28.449929 23299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:47:28.449993 23302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:47:28.449956 23300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:47:28.451411 23294 server_base.cc:1034] running on GCE node
I20260929 16:47:28.452384 23294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:47:28.454080 23294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:47:28.455332 23294 hybrid_clock.cc:648] HybridClock initialized: now 1790700448455237 us; error 73 us; skew 500 ppm
I20260929 16:47:28.455746 23294 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:47:28 dist-test-slave-l31f krb5kdc[23207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700448, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:47:28.459388 23294 init.cc:377] Logged in from keytab as kudu/127.0.197.125@KRBTEST.COM (short username kudu)
I20260929 16:47:28.461145 23294 webserver.cc:460] Webserver started at http://127.0.197.125:40623/ using document root <none> and password file <none>
I20260929 16:47:28.461957 23294 fs_manager.cc:359] Metadata directory not provided
I20260929 16:47:28.462069 23294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:47:28.462426 23294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:47:28.465828 23294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/minicluster-data/master-1/data/instance:
uuid: "c0a2182bf88242e7926531653947752b"
format_stamp: "Formatted at 2026-09-29 16:47:28 on dist-test-slave-l31f"
server_key: "bdb4aef1c0541f078ddf6ffb8e8c74a6"
server_key_iv: "44f4e7717070559c6fe2d44fe73b391e"
server_key_version: "encryptionkey@0"
I20260929 16:47:28.466744 23294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/minicluster-data/master-1/wal/instance:
uuid: "c0a2182bf88242e7926531653947752b"
format_stamp: "Formatted at 2026-09-29 16:47:28 on dist-test-slave-l31f"
server_key: "bdb4aef1c0541f078ddf6ffb8e8c74a6"
server_key_iv: "44f4e7717070559c6fe2d44fe73b391e"
server_key_version: "encryptionkey@0"
I20260929 16:47:28.472853 23294 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260929 16:47:28.477051 23309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:47:28.478703 23294 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:47:28.479115 23294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/minicluster-data/master-1/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/minicluster-data/master-1/wal
uuid: "c0a2182bf88242e7926531653947752b"
format_stamp: "Formatted at 2026-09-29 16:47:28 on dist-test-slave-l31f"
server_key: "bdb4aef1c0541f078ddf6ffb8e8c74a6"
server_key_iv: "44f4e7717070559c6fe2d44fe73b391e"
server_key_version: "encryptionkey@0"
I20260929 16:47:28.479423 23294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:47:28.500988 23294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:47:28.505740 23294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:47:28.506067 23294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:47:28.525903 23294 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.125:41099
I20260929 16:47:28.526010 23360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.125:41099 every 8 connection(s)
I20260929 16:47:28.527477 23294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/minicluster-data/master-1/data/info.pb
I20260929 16:47:28.531649 23361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:47:28.537328 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 23294
I20260929 16:47:28.537544 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/minicluster-data/master-1/wal/instance
I20260929 16:47:28.537979 789 external_mini_cluster.cc:1428] Setting key 979e84dbea7e352da7f545d1a4a65e8c
I20260929 16:47:28.539129 23361 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33615 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:47:28 dist-test-slave-l31f krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700448, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.125@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.197.124@KRBTEST.COM; defaulting to no policy
I20260929 16:47:28.566182 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:28.541984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51792 (local address 127.0.197.126:33615)
0929 16:47:28.542679 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:28.542692 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:28.543980 (+ 1288us) server_negotiation.cc:407] Connection header received
0929 16:47:28.547274 (+ 3294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:28.547280 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:28.547400 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:28.547537 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:28.549878 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:28.550688 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:28.553310 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:28.553748 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:28.553858 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:28.558804 (+ 4946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:47:28.558810 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:47:28.558813 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:47:28.558854 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:47:28.562089 (+ 3235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:28.563353 (+ 1264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:28.563358 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:28.563360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:28.563506 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:28.564135 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:28.564139 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:28.564141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:28.564439 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:47:28.564569 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:28.565661 (+ 1092us) server_negotiation.cc:299] Negotiation successful
0929 16:47:28.565895 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":397,"threads_started":1}
I20260929 16:47:28.567502 23364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:28.542746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:33615 (local address 127.0.0.1:51792)
0929 16:47:28.543715 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:28.543770 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 16:47:28.546806 (+ 3036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:47:28.547843 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:47:28.547853 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:47:28.548707 (+ 854us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:47:28.549597 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:47:28.549627 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:47:28.550899 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:47:28.550908 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:47:28.553108 (+ 2200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:47:28.553132 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:47:28.554025 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:47:28.554034 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:47:28.554398 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:28.555296 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:47:28.555334 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:47:28.558491 (+ 3157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:47:28.562311 (+ 3820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:47:28.562325 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:47:28.562349 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:47:28.563190 (+ 841us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:47:28.563638 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:47:28.563644 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:47:28.563648 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:47:28.563965 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:47:28.564591 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:47:28.564604 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:47:28.565166 (+ 562us) client_negotiation.cc:769] Sending connection context
0929 16:47:28.565504 (+ 338us) client_negotiation.cc:240] Negotiation successful
0929 16:47:28.566204 (+ 700us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":392,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.124@KRBTEST.COM" created.
I20260929 16:47:28.569778 23361 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.197.125" port: 41099 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:47:28 dist-test-slave-l31f krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700448, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.125@KRBTEST.COM for kudu/127.0.197.125@KRBTEST.COM
I20260929 16:47:28.588724 23366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:28.570988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44858 (local address 127.0.197.125:41099)
0929 16:47:28.571743 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:28.571750 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:28.571777 (+ 27us) server_negotiation.cc:407] Connection header received
0929 16:47:28.571833 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:28.571838 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:28.572033 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:28.572156 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:28.573286 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:28.574436 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:28.576414 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:28.577043 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:28.577155 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:28.581258 (+ 4103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:47:28.581315 (+ 57us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:47:28.581339 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:47:28.581393 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:47:28.585316 (+ 3923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:28.586294 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:28.586302 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:28.586304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:28.586433 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:28.586896 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:28.586899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:28.586901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:28.587360 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:47:28.587571 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:28.588030 (+ 459us) server_negotiation.cc:299] Negotiation successful
0929 16:47:28.588379 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":122,"server-negotiator.queue_time_us":598,"thread_start_us":395,"threads_started":1}
I20260929 16:47:28.588827 23364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:28.570749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.125:41099 (local address 127.0.0.1:44858)
0929 16:47:28.571077 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:28.571101 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 16:47:28.571236 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:47:28.572222 (+ 986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:47:28.572230 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:47:28.572769 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:47:28.573055 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:47:28.573071 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:47:28.574603 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:47:28.574611 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:47:28.576213 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:47:28.576233 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:47:28.577244 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:47:28.577256 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:47:28.577498 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:28.578240 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:47:28.578271 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:47:28.580889 (+ 2618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:47:28.585501 (+ 4612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:47:28.585510 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:47:28.585517 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:47:28.586119 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:47:28.586555 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:47:28.586563 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:47:28.586569 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:47:28.586759 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:47:28.587521 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:47:28.587529 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:47:28.587703 (+ 174us) client_negotiation.cc:769] Sending connection context
0929 16:47:28.587868 (+ 165us) client_negotiation.cc:240] Negotiation successful
0929 16:47:28.588249 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 16:47:28.591467 23361 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43025 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:47:28.592897 23364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:28.592264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43025 (local address 127.0.0.1:34018)
0929 16:47:28.592503 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:28.592661 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260929 16:47:28.593184 23310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43025: connect: Connection refused (error 111)
I20260929 16:47:28.594985 23364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:28.594497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43025 (local address 127.0.0.1:34022)
0929 16:47:28.594694 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:28.594818 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260929 16:47:28.595263 23361 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.197.124:43025: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43025: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/krb5kdc/kudu_127.0.197.124.keytab.
Entry for principal kudu/127.0.197.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/krb5kdc/kudu_127.0.197.124.keytab.
I20260929 16:47:28.606904 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.124:43025
--webserver_interface=127.0.197.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/krb5kdc/kudu_127.0.197.124.keytab
--principal=kudu/127.0.197.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:43335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.126:33615,127.0.197.125:41099,127.0.197.124:43025
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:47:28.644645 23361 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43025 } attempt: 1
I20260929 16:47:28.646013 23364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:28.645446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43025 (local address 127.0.0.1:34026)
0929 16:47:28.645680 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:28.645808 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260929 16:47:28.647421 23364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:28.646742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43025 (local address 127.0.0.1:34036)
0929 16:47:28.647100 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:28.647243 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
W20260929 16:47:28.647687 23361 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.197.124:43025: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43025: connect: Connection refused (error 111)
I20260929 16:47:28.699491 23287 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.197.125" port: 41099 } attempt: 5
Sep 29 16:47:28 dist-test-slave-l31f krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for kudu/127.0.197.125@KRBTEST.COM
I20260929 16:47:28.715581 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:28.700324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.125:41099 (local address 127.0.0.1:44868)
0929 16:47:28.700538 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:28.700561 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 16:47:28.700680 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:47:28.701108 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:47:28.701112 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:47:28.701502 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:47:28.701694 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:47:28.701704 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:47:28.703385 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:47:28.703390 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:47:28.704315 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:47:28.704330 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:47:28.705336 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:47:28.705341 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:47:28.705513 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:28.706272 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:47:28.706292 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:47:28.708876 (+ 2584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:47:28.712764 (+ 3888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:47:28.712770 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:47:28.712775 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:47:28.713226 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:47:28.713679 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:47:28.713685 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:47:28.713689 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:47:28.713844 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:47:28.714436 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:47:28.714441 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:47:28.714623 (+ 182us) client_negotiation.cc:769] Sending connection context
0929 16:47:28.714808 (+ 185us) client_negotiation.cc:240] Negotiation successful
0929 16:47:28.715209 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 16:47:28.715597 23366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:28.700453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44868 (local address 127.0.197.125:41099)
0929 16:47:28.700656 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:28.700662 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:28.700687 (+ 25us) server_negotiation.cc:407] Connection header received
0929 16:47:28.700800 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:28.700805 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:28.700892 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:28.701028 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:28.701879 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:28.703196 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:28.704503 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:28.705145 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:28.705287 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:28.709187 (+ 3900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:47:28.709194 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:47:28.709197 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:47:28.709238 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:47:28.712562 (+ 3324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:28.713380 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:28.713388 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:28.713396 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:28.713541 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:28.713965 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:28.713971 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:28.713974 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:28.714296 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:47:28.714478 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:28.714927 (+ 449us) server_negotiation.cc:299] Negotiation successful
0929 16:47:28.715209 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 16:47:28.715955 23361 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43025 } attempt: 2
I20260929 16:47:28.717054 23287 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43025 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:47:28.717135 23364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:28.716615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43025 (local address 127.0.0.1:34042)
0929 16:47:28.716827 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:28.716952 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 16:47:28.718122 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:28.717728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43025 (local address 127.0.0.1:34044)
0929 16:47:28.717891 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:28.717976 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 16:47:28.718223 23364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:28.717795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43025 (local address 127.0.0.1:34056)
0929 16:47:28.717991 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:28.718085 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20260929 16:47:28.718425 23361 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.197.124:43025: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43025: connect: Connection refused (error 111)
I20260929 16:47:28.719204 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:28.718781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43025 (local address 127.0.0.1:34058)
0929 16:47:28.718937 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:28.719091 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 16:47:28.719434 23287 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.197.124:43025: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43025: connect: Connection refused (error 111)
I20260929 16:47:28.770709 23287 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43025 } attempt: 1
I20260929 16:47:28.772181 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:28.771619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43025 (local address 127.0.0.1:34068)
0929 16:47:28.771861 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:28.771995 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260929 16:47:28.773290 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:28.772919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43025 (local address 127.0.0.1:34084)
0929 16:47:28.773076 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:28.773161 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 16:47:28.773520 23287 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.197.124:43025: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43025: connect: Connection refused (error 111)
I20260929 16:47:28.809687 23361 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43025 } attempt: 3
I20260929 16:47:28.811223 23364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:28.810459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43025 (local address 127.0.0.1:34096)
0929 16:47:28.810768 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:28.810911 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260929 16:47:28.812685 23364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:28.812118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43025 (local address 127.0.0.1:34102)
0929 16:47:28.812411 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:28.812522 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
W20260929 16:47:28.812985 23361 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.197.124:43025: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43025: connect: Connection refused (error 111)
I20260929 16:47:28.841776 23287 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43025 } attempt: 2
I20260929 16:47:28.843144 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:28.842529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43025 (local address 127.0.0.1:34116)
0929 16:47:28.842786 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:28.842897 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260929 16:47:28.844563 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:28.844157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43025 (local address 127.0.0.1:34126)
0929 16:47:28.844341 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:28.844431 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260929 16:47:28.844787 23287 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.197.124:43025: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43025: connect: Connection refused (error 111)
I20260929 16:47:28.951107 23287 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43025 } attempt: 3
I20260929 16:47:28.952271 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:28.951787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43025 (local address 127.0.0.1:34132)
0929 16:47:28.952008 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:28.952104 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260929 16:47:28.953400 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:28.952973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43025 (local address 127.0.0.1:34134)
0929 16:47:28.953206 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:28.953277 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260929 16:47:28.953605 23287 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.197.124:43025: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43025: connect: Connection refused (error 111)
I20260929 16:47:28.956264 23361 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43025 } attempt: 4
I20260929 16:47:28.957522 23364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:28.956936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43025 (local address 127.0.0.1:34144)
0929 16:47:28.957207 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:28.957325 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 16:47:28.958695 23364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:28.958278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43025 (local address 127.0.0.1:34156)
0929 16:47:28.958457 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:28.958550 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260929 16:47:28.958911 23361 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.197.124:43025: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43025: connect: Connection refused (error 111)
W20260929 16:47:28.973840 23369 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 16:47:28.974191 23369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:47:28.974257 23369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:47:28.974356 23369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:47:28.982106 23369 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:47:28.982224 23369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:47:28.982282 23369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:47:28.982333 23369 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:47:28.982396 23369 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:47:28.993824 23369 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:43335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.197.126:33615,127.0.197.125:41099,127.0.197.124:43025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/krb5kdc/kudu_127.0.197.124.keytab
--principal=kudu/127.0.197.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.124:43025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.197.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:47:28.995654 23369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:47:28.997999 23369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:47:29.012701 23374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:47:29.012676 23375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:47:29.012753 23377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:47:29.014001 23369 server_base.cc:1034] running on GCE node
I20260929 16:47:29.014957 23369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:47:29.016654 23369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:47:29.017935 23369 hybrid_clock.cc:648] HybridClock initialized: now 1790700449017865 us; error 47 us; skew 500 ppm
I20260929 16:47:29.018311 23369 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:47:29 dist-test-slave-l31f krb5kdc[23207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:47:29.029642 23369 init.cc:377] Logged in from keytab as kudu/127.0.197.124@KRBTEST.COM (short username kudu)
I20260929 16:47:29.031454 23369 webserver.cc:460] Webserver started at http://127.0.197.124:44287/ using document root <none> and password file <none>
I20260929 16:47:29.032253 23369 fs_manager.cc:359] Metadata directory not provided
I20260929 16:47:29.032408 23369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:47:29.032756 23369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:47:29.036203 23369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/minicluster-data/master-2/data/instance:
uuid: "1be9b47795394e6293194b4b30b01663"
format_stamp: "Formatted at 2026-09-29 16:47:29 on dist-test-slave-l31f"
server_key: "3e01219646dfeafb7b483aefbeae14b7"
server_key_iv: "c7df922178be405e3aec4362189daa0a"
server_key_version: "encryptionkey@0"
I20260929 16:47:29.037127 23369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/minicluster-data/master-2/wal/instance:
uuid: "1be9b47795394e6293194b4b30b01663"
format_stamp: "Formatted at 2026-09-29 16:47:29 on dist-test-slave-l31f"
server_key: "3e01219646dfeafb7b483aefbeae14b7"
server_key_iv: "c7df922178be405e3aec4362189daa0a"
server_key_version: "encryptionkey@0"
I20260929 16:47:29.042896 23369 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260929 16:47:29.047147 23384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:47:29.048786 23369 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 16:47:29.049036 23369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/minicluster-data/master-2/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/minicluster-data/master-2/wal
uuid: "1be9b47795394e6293194b4b30b01663"
format_stamp: "Formatted at 2026-09-29 16:47:29 on dist-test-slave-l31f"
server_key: "3e01219646dfeafb7b483aefbeae14b7"
server_key_iv: "c7df922178be405e3aec4362189daa0a"
server_key_version: "encryptionkey@0"
I20260929 16:47:29.049300 23369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:47:29.082610 23369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:47:29.087881 23369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:47:29.088194 23369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:47:29.111315 23369 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.124:43025
I20260929 16:47:29.111330 23435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.124:43025 every 8 connection(s)
I20260929 16:47:29.112838 23369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/minicluster-data/master-2/data/info.pb
I20260929 16:47:29.116393 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 23369
I20260929 16:47:29.116622 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790700068255611-789-0/minicluster-data/master-2/wal/instance
I20260929 16:47:29.117065 789 external_mini_cluster.cc:1428] Setting key 142b0bbc6cf5c0d1516210c594843e9d
I20260929 16:47:29.117059 23436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:47:29.126566 23436 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33615 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:47:29.131161 23287 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43025 } attempt: 4
Sep 29 16:47:29 dist-test-slave-l31f krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.124@KRBTEST.COM
Sep 29 16:47:29 dist-test-slave-l31f krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.124@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
Sep 29 16:47:29 dist-test-slave-l31f krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for kudu/127.0.197.124@KRBTEST.COM
I20260929 16:47:29.145761 23439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:29.119372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34166 (local address 127.0.197.124:43025)
0929 16:47:29.120218 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:29.120234 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:29.120280 (+ 46us) server_negotiation.cc:407] Connection header received
0929 16:47:29.122077 (+ 1797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:29.122105 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:29.122600 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:29.123175 (+ 575us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:29.124783 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:29.126079 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:29.127626 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:29.128414 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:29.128565 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:29.134680 (+ 6115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:47:29.134716 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:47:29.134744 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:47:29.134831 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:47:29.140055 (+ 5224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:29.141066 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:29.141079 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:29.141090 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:29.141293 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:29.141794 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:29.141799 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:29.141802 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:29.142603 (+ 801us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:47:29.143040 (+ 437us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:29.143530 (+ 490us) server_negotiation.cc:299] Negotiation successful
0929 16:47:29.144072 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":368,"threads_started":1}
I20260929 16:47:29.149740 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:29.131790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43025 (local address 127.0.0.1:34172)
0929 16:47:29.132065 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:29.132086 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:47:29.132199 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:47:29.133064 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:47:29.133071 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:47:29.133616 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:47:29.133855 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:47:29.133867 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:47:29.135779 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:47:29.135788 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:47:29.137590 (+ 1802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:47:29.137610 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:47:29.138740 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:47:29.138747 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:47:29.138992 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:29.139647 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:47:29.139672 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:47:29.142269 (+ 2597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:47:29.146759 (+ 4490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:47:29.146768 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:47:29.146774 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:47:29.147375 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:47:29.147825 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:47:29.147830 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:47:29.147836 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:47:29.148016 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:47:29.148667 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:47:29.148674 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:47:29.148838 (+ 164us) client_negotiation.cc:769] Sending connection context
0929 16:47:29.148997 (+ 159us) client_negotiation.cc:240] Negotiation successful
0929 16:47:29.149300 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":25}
I20260929 16:47:29.149822 23442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:29.131945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34172 (local address 127.0.197.124:43025)
0929 16:47:29.132774 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:29.132778 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:29.132795 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:47:29.132836 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:29.132840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:29.132895 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:29.133031 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:29.134048 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:29.135576 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:29.137796 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:29.138560 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:29.138711 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:29.142508 (+ 3797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:47:29.142513 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:47:29.142516 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:47:29.142550 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:47:29.146618 (+ 4068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:29.147511 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:29.147517 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:29.147520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:29.147701 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:29.148147 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:29.148151 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:29.148154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:29.148516 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:47:29.148656 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:29.149113 (+ 457us) server_negotiation.cc:299] Negotiation successful
0929 16:47:29.149395 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":715,"spinlock_wait_cycles":29952,"thread_start_us":550,"threads_started":1}
I20260929 16:47:29.150311 23440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:29.129517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:33615 (local address 127.0.0.1:51804)
0929 16:47:29.130456 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:29.130524 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0929 16:47:29.130728 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:47:29.131875 (+ 1147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:47:29.131884 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:47:29.132990 (+ 1106us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:47:29.133505 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:47:29.133524 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:47:29.135292 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:47:29.135298 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:47:29.136830 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:47:29.136843 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:47:29.137747 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:47:29.137755 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:47:29.138004 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:29.138734 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:47:29.138767 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:47:29.141971 (+ 3204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:47:29.146184 (+ 4213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:47:29.146194 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:47:29.146202 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:47:29.146951 (+ 749us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:47:29.147453 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:47:29.147458 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:47:29.147461 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:47:29.147690 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:47:29.148515 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:47:29.148524 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:47:29.148710 (+ 186us) client_negotiation.cc:769] Sending connection context
0929 16:47:29.148906 (+ 196us) client_negotiation.cc:240] Negotiation successful
0929 16:47:29.149259 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"spinlock_wait_cycles":21760,"thread_start_us":424,"threads_started":1}
I20260929 16:47:29.150436 23441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:29.129737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51804 (local address 127.0.197.126:33615)
0929 16:47:29.131378 (+ 1641us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:29.131386 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:29.131410 (+ 24us) server_negotiation.cc:407] Connection header received
0929 16:47:29.131481 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:29.131489 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:29.131613 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:29.133427 (+ 1814us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:29.133684 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:29.134993 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:29.137035 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:29.137536 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:29.137988 (+ 452us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:29.142208 (+ 4220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:47:29.142214 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:47:29.142218 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:47:29.142257 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:47:29.145978 (+ 3721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:29.147116 (+ 1138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:29.147122 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:29.147125 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:29.147335 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:29.147839 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:29.147844 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:29.147848 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:29.148351 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:47:29.149543 (+ 1192us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:29.149669 (+ 126us) server_negotiation.cc:299] Negotiation successful
0929 16:47:29.149888 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":1474,"spinlock_wait_cycles":16128,"thread_start_us":1260,"threads_started":1}
I20260929 16:47:29.152729 23436 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.197.125" port: 41099 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:47:29.157348 23287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57843dd206c24c658e60f97672cfc60c: Bootstrap starting.
Sep 29 16:47:29 dist-test-slave-l31f krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:47:29.161289 23287 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57843dd206c24c658e60f97672cfc60c: Neither blocks nor log segments found. Creating new log.
Sep 29 16:47:29 dist-test-slave-l31f krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.124@KRBTEST.COM for kudu/127.0.197.125@KRBTEST.COM
I20260929 16:47:29.162892 23287 log.cc:824] T 00000000000000000000000000000000 P 57843dd206c24c658e60f97672cfc60c: Log is configured to *not* fsync() on all Append() calls
I20260929 16:47:29.166400 23287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57843dd206c24c658e60f97672cfc60c: No bootstrap required, opened a new log
I20260929 16:47:29.170610 23366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:29.153644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44874 (local address 127.0.197.125:41099)
0929 16:47:29.153938 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:29.153946 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:29.154146 (+ 200us) server_negotiation.cc:407] Connection header received
0929 16:47:29.154307 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:29.154314 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:29.154393 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:29.154494 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:29.155494 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:29.156547 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:29.158261 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:29.159080 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:29.159232 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:29.163847 (+ 4615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:47:29.163855 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:47:29.163860 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:47:29.163912 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:47:29.167449 (+ 3537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:29.168372 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:29.168379 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:29.168382 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:29.168528 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:29.168932 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:29.168937 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:29.168941 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:29.169329 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:47:29.169497 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:29.169954 (+ 457us) server_negotiation.cc:299] Negotiation successful
0929 16:47:29.170216 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20260929 16:47:29.171089 23441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:29.150016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51810 (local address 127.0.197.126:33615)
0929 16:47:29.150903 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:29.150911 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:29.151972 (+ 1061us) server_negotiation.cc:407] Connection header received
0929 16:47:29.152034 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:29.152039 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:29.152108 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:29.152221 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:29.153847 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:29.154878 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:29.156240 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:29.156695 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:29.156842 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:29.162387 (+ 5545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:47:29.162397 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:47:29.162400 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:47:29.162454 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:47:29.166761 (+ 4307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:29.167786 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:29.167792 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:29.167795 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:29.167940 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:29.168393 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:29.168399 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:29.168405 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:29.168843 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:47:29.169268 (+ 425us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:29.170403 (+ 1135us) server_negotiation.cc:299] Negotiation successful
0929 16:47:29.170702 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":714,"thread_start_us":1450,"threads_started":1}
I20260929 16:47:29.171479 23440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:29.153669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.125:41099 (local address 127.0.0.1:44874)
0929 16:47:29.154044 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:29.154063 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:47:29.154184 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:47:29.154542 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:47:29.154548 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:47:29.155112 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:47:29.155305 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:47:29.155316 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:47:29.156760 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:47:29.156765 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:47:29.158080 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:47:29.158096 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:47:29.159486 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:47:29.159495 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:47:29.159700 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:29.160424 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:47:29.160449 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:47:29.163535 (+ 3086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:47:29.167615 (+ 4080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:47:29.167621 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:47:29.167625 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:47:29.168237 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:47:29.168651 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:47:29.168656 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:47:29.168660 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:47:29.168812 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:47:29.169471 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:47:29.169477 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:47:29.169657 (+ 180us) client_negotiation.cc:769] Sending connection context
0929 16:47:29.169827 (+ 170us) client_negotiation.cc:240] Negotiation successful
0929 16:47:29.170503 (+ 676us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238}
I20260929 16:47:29.172760 23436 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43025 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:47:29.178741 23287 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57843dd206c24c658e60f97672cfc60c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57843dd206c24c658e60f97672cfc60c" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33615 } } peers { permanent_uuid: "c0a2182bf88242e7926531653947752b" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 41099 } } peers { permanent_uuid: "1be9b47795394e6293194b4b30b01663" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43025 } }
I20260929 16:47:29.179255 23287 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57843dd206c24c658e60f97672cfc60c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57843dd206c24c658e60f97672cfc60c, State: Initialized, Role: FOLLOWER
I20260929 16:47:29.179870 23287 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57843dd206c24c658e60f97672cfc60c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57843dd206c24c658e60f97672cfc60c" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33615 } } peers { permanent_uuid: "c0a2182bf88242e7926531653947752b" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 41099 } } peers { permanent_uuid: "1be9b47795394e6293194b4b30b01663" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43025 } }
I20260929 16:47:29.181010 23445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57843dd206c24c658e60f97672cfc60c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57843dd206c24c658e60f97672cfc60c" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33615 } } peers { permanent_uuid: "c0a2182bf88242e7926531653947752b" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 41099 } } peers { permanent_uuid: "1be9b47795394e6293194b4b30b01663" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43025 } } }
I20260929 16:47:29.181495 23445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57843dd206c24c658e60f97672cfc60c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:47:29.182407 23287 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57843dd206c24c658e60f97672cfc60c [sys.catalog]: configured and running, proceeding with master startup.
Sep 29 16:47:29 dist-test-slave-l31f krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.124@KRBTEST.COM for kudu/127.0.197.124@KRBTEST.COM
Sep 29 16:47:29 dist-test-slave-l31f krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.125@KRBTEST.COM
I20260929 16:47:29.191740 23440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:29.173566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43025 (local address 127.0.0.1:34180)
0929 16:47:29.174342 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:29.174362 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:47:29.174476 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:47:29.174884 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:47:29.174888 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:47:29.175416 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:47:29.175614 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:47:29.175627 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:47:29.176999 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:47:29.177004 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:47:29.178178 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:47:29.178192 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:47:29.179142 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:47:29.179148 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:47:29.179602 (+ 454us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:29.180364 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:47:29.180394 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:47:29.183785 (+ 3391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:47:29.188462 (+ 4677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:47:29.188469 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:47:29.188475 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:47:29.189050 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:47:29.189501 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:47:29.189505 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:47:29.189508 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:47:29.189680 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:47:29.190400 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:47:29.190404 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:47:29.190577 (+ 173us) client_negotiation.cc:769] Sending connection context
0929 16:47:29.190736 (+ 159us) client_negotiation.cc:240] Negotiation successful
0929 16:47:29.191134 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"mutex_wait_us":3}
I20260929 16:47:29.192747 23366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:29.173281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44878 (local address 127.0.197.125:41099)
0929 16:47:29.173678 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:29.173684 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:29.173704 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:47:29.173759 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:29.173764 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:29.173856 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:29.173961 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:29.175525 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:29.176261 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:29.177798 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:29.178437 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:29.178582 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:29.184406 (+ 5824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:47:29.184413 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:47:29.184418 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:47:29.184467 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:47:29.188302 (+ 3835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:29.190055 (+ 1753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:29.190061 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:29.190065 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:29.190270 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:29.190815 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:29.190820 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:29.190824 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:29.191218 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:47:29.191499 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:29.192113 (+ 614us) server_negotiation.cc:299] Negotiation successful
0929 16:47:29.192382 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 16:47:29.193200 23442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:29.173523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34180 (local address 127.0.197.124:43025)
0929 16:47:29.174240 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:29.174246 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:29.174452 (+ 206us) server_negotiation.cc:407] Connection header received
0929 16:47:29.174619 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:29.174628 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:29.174714 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:29.174832 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:29.175776 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:29.176841 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:29.178336 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:29.178906 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:29.179125 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:29.184072 (+ 4947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:47:29.184082 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:47:29.184090 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:47:29.184139 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:47:29.188290 (+ 4151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:29.189203 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:29.189209 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:29.189213 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:29.189371 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:29.189796 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:29.189801 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:29.189803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:29.190182 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:47:29.190345 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:29.192630 (+ 2285us) server_negotiation.cc:299] Negotiation successful
0929 16:47:29.192884 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":575}
I20260929 16:47:29.199218 789 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:47:29.200734 23436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1be9b47795394e6293194b4b30b01663: Bootstrap starting.
I20260929 16:47:29.203226 23262 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "57843dd206c24c658e60f97672cfc60c"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51810
I20260929 16:47:29.203717 23262 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57843dd206c24c658e60f97672cfc60c [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 16:47:29.203934 23262 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57843dd206c24c658e60f97672cfc60c [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:47:29.205147 23436 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1be9b47795394e6293194b4b30b01663: Neither blocks nor log segments found. Creating new log.
I20260929 16:47:29.206620 23436 log.cc:824] T 00000000000000000000000000000000 P 1be9b47795394e6293194b4b30b01663: Log is configured to *not* fsync() on all Append() calls
I20260929 16:47:29.207682 23262 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57843dd206c24c658e60f97672cfc60c [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57843dd206c24c658e60f97672cfc60c" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33615 } } peers { permanent_uuid: "c0a2182bf88242e7926531653947752b" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 41099 } } peers { permanent_uuid: "1be9b47795394e6293194b4b30b01663" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43025 } }
W20260929 16:47:29.209254 23456 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 57843dd206c24c658e60f97672cfc60c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:47:29.209403 23456 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:47:29.210187 23436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1be9b47795394e6293194b4b30b01663: No bootstrap required, opened a new log
I20260929 16:47:29.210860 23262 leader_election.cc:288] T 00000000000000000000000000000000 P 57843dd206c24c658e60f97672cfc60c [CANDIDATE]: Term 1 election: Requested vote from peers c0a2182bf88242e7926531653947752b (127.0.197.125:41099), 1be9b47795394e6293194b4b30b01663 (127.0.197.124:43025)
I20260929 16:47:29.194581 23220 master_runner.cc:186] Error getting master registration for 127.0.197.125:41099: OK, instance_id { permanent_uuid: "c0a2182bf88242e7926531653947752b" instance_seqno: 1790700448520103 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 16:47:29.215293 23220 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 16:47:29.219930 23241 tablet.cc:2369] T 00000000000000000000000000000000 P 57843dd206c24c658e60f97672cfc60c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:47:29.224216 23436 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1be9b47795394e6293194b4b30b01663 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57843dd206c24c658e60f97672cfc60c" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33615 } } peers { permanent_uuid: "c0a2182bf88242e7926531653947752b" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 41099 } } peers { permanent_uuid: "1be9b47795394e6293194b4b30b01663" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43025 } }
I20260929 16:47:29.224767 23436 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1be9b47795394e6293194b4b30b01663 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1be9b47795394e6293194b4b30b01663, State: Initialized, Role: FOLLOWER
I20260929 16:47:29.225405 23436 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1be9b47795394e6293194b4b30b01663 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57843dd206c24c658e60f97672cfc60c" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33615 } } peers { permanent_uuid: "c0a2182bf88242e7926531653947752b" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 41099 } } peers { permanent_uuid: "1be9b47795394e6293194b4b30b01663" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43025 } }
I20260929 16:47:29.239782 23462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1be9b47795394e6293194b4b30b01663 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57843dd206c24c658e60f97672cfc60c" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33615 } } peers { permanent_uuid: "c0a2182bf88242e7926531653947752b" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 41099 } } peers { permanent_uuid: "1be9b47795394e6293194b4b30b01663" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43025 } } }
I20260929 16:47:29.240414 23462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1be9b47795394e6293194b4b30b01663 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:47:29.241410 23436 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1be9b47795394e6293194b4b30b01663 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:47:29.241473 23411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "57843dd206c24c658e60f97672cfc60c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1be9b47795394e6293194b4b30b01663"
I20260929 16:47:29.242069 23411 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1be9b47795394e6293194b4b30b01663 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:47:29.247152 23411 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1be9b47795394e6293194b4b30b01663 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57843dd206c24c658e60f97672cfc60c in term 1.
I20260929 16:47:29.248405 23236 leader_election.cc:302] T 00000000000000000000000000000000 P 57843dd206c24c658e60f97672cfc60c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1be9b47795394e6293194b4b30b01663, 57843dd206c24c658e60f97672cfc60c; no voters:
I20260929 16:47:29.249083 23445 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57843dd206c24c658e60f97672cfc60c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:47:29.258404 23361 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43025 } attempt: 5
I20260929 16:47:29.268477 23445 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57843dd206c24c658e60f97672cfc60c [term 1 LEADER]: Becoming Leader. State: Replica: 57843dd206c24c658e60f97672cfc60c, State: Running, Role: LEADER
I20260929 16:47:29.269189 23445 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57843dd206c24c658e60f97672cfc60c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57843dd206c24c658e60f97672cfc60c" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33615 } } peers { permanent_uuid: "c0a2182bf88242e7926531653947752b" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 41099 } } peers { permanent_uuid: "1be9b47795394e6293194b4b30b01663" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43025 } }
Sep 29 16:47:29 dist-test-slave-l31f krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700448, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.125@KRBTEST.COM for kudu/127.0.197.124@KRBTEST.COM
I20260929 16:47:29.285482 23364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:29.259219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43025 (local address 127.0.0.1:34200)
0929 16:47:29.259474 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:29.259502 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 16:47:29.259651 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:47:29.260232 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:47:29.260239 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:47:29.260990 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:47:29.261289 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:47:29.261306 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:47:29.263204 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:47:29.263209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:47:29.264678 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:47:29.264698 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:47:29.266415 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:47:29.266425 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:47:29.266750 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:29.267744 (+ 994us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:47:29.267780 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:47:29.271169 (+ 3389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:47:29.280899 (+ 9730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:47:29.280908 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:47:29.280915 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:47:29.281665 (+ 750us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:47:29.282572 (+ 907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:47:29.282591 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:47:29.282595 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:47:29.282780 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:47:29.284024 (+ 1244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:47:29.284033 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:47:29.284268 (+ 235us) client_negotiation.cc:769] Sending connection context
0929 16:47:29.284442 (+ 174us) client_negotiation.cc:240] Negotiation successful
0929 16:47:29.284882 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 16:47:29.288357 23439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:29.259595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34200 (local address 127.0.197.124:43025)
0929 16:47:29.259804 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:29.259812 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:29.259840 (+ 28us) server_negotiation.cc:407] Connection header received
0929 16:47:29.259921 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:29.259930 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:29.260034 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:29.260174 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:29.261511 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:29.262965 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:29.264898 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:29.265876 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:29.266063 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:29.271562 (+ 5499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:47:29.271577 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:47:29.271585 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:47:29.271724 (+ 139us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:47:29.280558 (+ 8834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:29.282024 (+ 1466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:29.282035 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:29.282039 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:29.282336 (+ 297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:29.283067 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:29.283099 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:29.283103 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:29.283769 (+ 666us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:47:29.286477 (+ 2708us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:29.286799 (+ 322us) server_negotiation.cc:299] Negotiation successful
0929 16:47:29.287410 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 16:47:29.293467 23366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:29.230633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44884 (local address 127.0.197.125:41099)
0929 16:47:29.230822 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:29.230828 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:29.244581 (+ 13753us) server_negotiation.cc:407] Connection header received
0929 16:47:29.245015 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:29.245023 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:29.245116 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:29.245197 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:29.249509 (+ 4312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:29.251056 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:29.255832 (+ 4776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:29.256567 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:29.256686 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:29.274615 (+ 17929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:47:29.274623 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:47:29.274628 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:47:29.274675 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:47:29.278800 (+ 4125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:29.284931 (+ 6131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:29.284937 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:29.284940 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:29.285176 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:29.288442 (+ 3266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:29.288452 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:29.288458 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:29.288988 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:47:29.289248 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:29.292614 (+ 3366us) server_negotiation.cc:299] Negotiation successful
0929 16:47:29.293019 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 16:47:29.298806 23361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0a2182bf88242e7926531653947752b: Bootstrap starting.
I20260929 16:47:29.308853 23361 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0a2182bf88242e7926531653947752b: Neither blocks nor log segments found. Creating new log.
I20260929 16:47:29.310999 23442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:29.240886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34190 (local address 127.0.197.124:43025)
0929 16:47:29.241071 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:29.241077 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:29.255697 (+ 14620us) server_negotiation.cc:407] Connection header received
0929 16:47:29.256296 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:29.256307 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:29.256448 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:29.256551 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:29.271593 (+ 15042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:29.272670 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:29.282236 (+ 9566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:29.283157 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:29.283317 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:29.301735 (+ 18418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:47:29.301743 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:47:29.301750 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:47:29.301802 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:47:29.305919 (+ 4117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:29.307345 (+ 1426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:29.307356 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:29.307362 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:29.307632 (+ 270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:29.308249 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:29.308259 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:29.308265 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:29.308775 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:47:29.309206 (+ 431us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:29.310016 (+ 810us) server_negotiation.cc:299] Negotiation successful
0929 16:47:29.310378 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69,"spinlock_wait_cycles":5888}
I20260929 16:47:29.313759 23361 log.cc:824] T 00000000000000000000000000000000 P c0a2182bf88242e7926531653947752b: Log is configured to *not* fsync() on all Append() calls
I20260929 16:47:29.320624 23441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:29.230208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51816 (local address 127.0.197.126:33615)
0929 16:47:29.230487 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:29.230498 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:29.249827 (+ 19329us) server_negotiation.cc:407] Connection header received
0929 16:47:29.250161 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:29.250170 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:29.250311 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:29.250404 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:29.268362 (+ 17958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:29.269487 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:29.280198 (+ 10711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:29.280956 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:29.281085 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:29.295837 (+ 14752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:47:29.295844 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:47:29.295849 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:47:29.295895 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:47:29.300586 (+ 4691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:29.307083 (+ 6497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:29.307094 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:29.307101 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:29.314872 (+ 7771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:29.318381 (+ 3509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:29.318388 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:29.318392 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:29.318881 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:47:29.319218 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:29.319892 (+ 674us) server_negotiation.cc:299] Negotiation successful
0929 16:47:29.320219 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260929 16:47:29.325853 23468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57843dd206c24c658e60f97672cfc60c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57843dd206c24c658e60f97672cfc60c. Latest consensus state: current_term: 1 leader_uuid: "57843dd206c24c658e60f97672cfc60c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57843dd206c24c658e60f97672cfc60c" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33615 } } peers { permanent_uuid: "c0a2182bf88242e7926531653947752b" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 41099 } } peers { permanent_uuid: "1be9b47795394e6293194b4b30b01663" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43025 } } }
I20260929 16:47:29.326639 23468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57843dd206c24c658e60f97672cfc60c [sys.catalog]: This master's current role is: LEADER
I20260929 16:47:29.328164 23480 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:47:29.331584 23361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0a2182bf88242e7926531653947752b: No bootstrap required, opened a new log
I20260929 16:47:29.336102 23481 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260929 16:47:29.340488 23390 tablet.cc:2369] T 00000000000000000000000000000000 P 1be9b47795394e6293194b4b30b01663: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:47:29.343813 23361 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0a2182bf88242e7926531653947752b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57843dd206c24c658e60f97672cfc60c" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33615 } } peers { permanent_uuid: "c0a2182bf88242e7926531653947752b" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 41099 } } peers { permanent_uuid: "1be9b47795394e6293194b4b30b01663" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43025 } }
I20260929 16:47:29.344185 23361 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0a2182bf88242e7926531653947752b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0a2182bf88242e7926531653947752b, State: Initialized, Role: FOLLOWER
I20260929 16:47:29.344695 23361 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0a2182bf88242e7926531653947752b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57843dd206c24c658e60f97672cfc60c" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33615 } } peers { permanent_uuid: "c0a2182bf88242e7926531653947752b" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 41099 } } peers { permanent_uuid: "1be9b47795394e6293194b4b30b01663" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43025 } }
I20260929 16:47:29.345057 23480 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:47:29.345672 23482 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0a2182bf88242e7926531653947752b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57843dd206c24c658e60f97672cfc60c" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33615 } } peers { permanent_uuid: "c0a2182bf88242e7926531653947752b" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 41099 } } peers { permanent_uuid: "1be9b47795394e6293194b4b30b01663" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43025 } } }
I20260929 16:47:29.346282 23482 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0a2182bf88242e7926531653947752b [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:47:29.347188 23361 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0a2182bf88242e7926531653947752b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:47:29.347820 23336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "57843dd206c24c658e60f97672cfc60c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c0a2182bf88242e7926531653947752b"
I20260929 16:47:29.348392 23336 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0a2182bf88242e7926531653947752b [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:47:29.352748 23336 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c0a2182bf88242e7926531653947752b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57843dd206c24c658e60f97672cfc60c in term 1.
I20260929 16:47:29.363721 23411 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1be9b47795394e6293194b4b30b01663 [term 1 FOLLOWER]: Refusing update from remote peer 57843dd206c24c658e60f97672cfc60c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:47:29.348853 23369 master_runner.cc:186] Error getting master registration for 127.0.197.125:41099: OK, instance_id { permanent_uuid: "c0a2182bf88242e7926531653947752b" instance_seqno: 1790700448520103 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 16:47:29.364560 23369 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 16:47:29.368173 23468 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 57843dd206c24c658e60f97672cfc60c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1be9b47795394e6293194b4b30b01663" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:47:29.370296 23336 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c0a2182bf88242e7926531653947752b [term 1 FOLLOWER]: Refusing update from remote peer 57843dd206c24c658e60f97672cfc60c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:47:29.373174 23468 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 57843dd206c24c658e60f97672cfc60c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0a2182bf88242e7926531653947752b" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 41099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:47:29.395228 23462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1be9b47795394e6293194b4b30b01663 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57843dd206c24c658e60f97672cfc60c. Latest consensus state: current_term: 1 leader_uuid: "57843dd206c24c658e60f97672cfc60c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57843dd206c24c658e60f97672cfc60c" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33615 } } peers { permanent_uuid: "c0a2182bf88242e7926531653947752b" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 41099 } } peers { permanent_uuid: "1be9b47795394e6293194b4b30b01663" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43025 } } }
I20260929 16:47:29.395592 23462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1be9b47795394e6293194b4b30b01663 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:47:29.399578 23484 mvcc.cc:205] Tried to move back new op lower bound from 7334709040579747840 to 7334709040220147712. Current Snapshot: MvccSnapshot[applied={T|T < 7334709040579747840}]
I20260929 16:47:29.402609 23462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1be9b47795394e6293194b4b30b01663 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "57843dd206c24c658e60f97672cfc60c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57843dd206c24c658e60f97672cfc60c" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33615 } } peers { permanent_uuid: "c0a2182bf88242e7926531653947752b" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 41099 } } peers { permanent_uuid: "1be9b47795394e6293194b4b30b01663" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43025 } } }
I20260929 16:47:29.402937 23462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1be9b47795394e6293194b4b30b01663 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:47:29.412907 23445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57843dd206c24c658e60f97672cfc60c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "57843dd206c24c658e60f97672cfc60c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57843dd206c24c658e60f97672cfc60c" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33615 } } peers { permanent_uuid: "c0a2182bf88242e7926531653947752b" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 41099 } } peers { permanent_uuid: "1be9b47795394e6293194b4b30b01663" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43025 } } }
I20260929 16:47:29.413295 23445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57843dd206c24c658e60f97672cfc60c [sys.catalog]: This master's current role is: LEADER
I20260929 16:47:29.413853 23480 catalog_manager.cc:1347] Generated new cluster ID: 18dddb1e795d4b19beb257e6bc420318
I20260929 16:47:29.413949 23480 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:47:29.421877 23491 mvcc.cc:205] Tried to move back new op lower bound from 7334709040579747840 to 7334709040220147712. Current Snapshot: MvccSnapshot[applied={T|T < 7334709040579747840 or (T in {7334709040579747840})}]
I20260929 16:47:29.431511 23480 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:47:29.433637 23480 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:47:29.436673 23482 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0a2182bf88242e7926531653947752b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57843dd206c24c658e60f97672cfc60c. Latest consensus state: current_term: 1 leader_uuid: "57843dd206c24c658e60f97672cfc60c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57843dd206c24c658e60f97672cfc60c" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33615 } } peers { permanent_uuid: "c0a2182bf88242e7926531653947752b" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 41099 } } peers { permanent_uuid: "1be9b47795394e6293194b4b30b01663" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43025 } } }
I20260929 16:47:29.437012 23482 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0a2182bf88242e7926531653947752b [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:47:29.445568 23492 mvcc.cc:205] Tried to move back new op lower bound from 7334709040579747840 to 7334709040220147712. Current Snapshot: MvccSnapshot[applied={T|T < 7334709040579747840}]
I20260929 16:47:29.447657 23468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57843dd206c24c658e60f97672cfc60c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "57843dd206c24c658e60f97672cfc60c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57843dd206c24c658e60f97672cfc60c" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33615 } } peers { permanent_uuid: "c0a2182bf88242e7926531653947752b" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 41099 } } peers { permanent_uuid: "1be9b47795394e6293194b4b30b01663" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43025 } } }
I20260929 16:47:29.448055 23468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57843dd206c24c658e60f97672cfc60c [sys.catalog]: This master's current role is: LEADER
I20260929 16:47:29.453342 23480 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57843dd206c24c658e60f97672cfc60c: Generated new TSK 0
I20260929 16:47:29.454752 23480 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:47:29.461503 23482 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0a2182bf88242e7926531653947752b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "57843dd206c24c658e60f97672cfc60c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57843dd206c24c658e60f97672cfc60c" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 33615 } } peers { permanent_uuid: "c0a2182bf88242e7926531653947752b" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 41099 } } peers { permanent_uuid: "1be9b47795394e6293194b4b30b01663" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43025 } } }
I20260929 16:47:29.461858 23482 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0a2182bf88242e7926531653947752b [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:47:29.478220 23503 catalog_manager.cc:1259] Loaded cluster ID: 18dddb1e795d4b19beb257e6bc420318
I20260929 16:47:29.478368 23503 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c0a2182bf88242e7926531653947752b: loading cluster ID for follower catalog manager: success
I20260929 16:47:29.482405 23503 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c0a2182bf88242e7926531653947752b: acquiring CA information for follower catalog manager: success
I20260929 16:47:29.485432 23503 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c0a2182bf88242e7926531653947752b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 16:47:29.568002 23366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:47:29.559128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44886 (local address 127.0.197.125:41099)
0929 16:47:29.559382 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:29.559389 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:29.565685 (+ 6296us) server_negotiation.cc:407] Connection header received
0929 16:47:29.566272 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:29.566279 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:29.566390 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:29.566478 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:29.567746 (+ 1268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44886: BlockingRecv error: recv got EOF from 127.0.0.1:44886 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260929 16:47:29.570217 23441 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:47:29.559495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51818 (local address 127.0.197.126:33615)
0929 16:47:29.559718 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:29.559725 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:29.565140 (+ 5415us) server_negotiation.cc:407] Connection header received
0929 16:47:29.568513 (+ 3373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:29.568519 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:29.568619 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:29.568707 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:29.569940 (+ 1233us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51818: BlockingRecv error: recv got EOF from 127.0.0.1:51818 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260929 16:47:29.573069 23442 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:47:29.565474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34216 (local address 127.0.197.124:43025)
0929 16:47:29.565709 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:29.565720 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:29.571279 (+ 5559us) server_negotiation.cc:407] Connection header received
0929 16:47:29.571655 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:29.571662 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:29.571754 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:29.571831 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:29.572821 (+ 990us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34216: BlockingRecv error: recv got EOF from 127.0.0.1:34216 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 16:47:29.588011 23366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:29.579305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44894 (local address 127.0.197.125:41099)
0929 16:47:29.579514 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:29.579521 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:29.580102 (+ 581us) server_negotiation.cc:407] Connection header received
0929 16:47:29.580345 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:29.580351 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:29.580589 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:29.580787 (+ 198us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:47:29.581270 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:29.582503 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:29.584847 (+ 2344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:29.585600 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:29.585744 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:29.586398 (+ 654us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:47:29.587255 (+ 857us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:47:29.587408 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:29.587529 (+ 121us) server_negotiation.cc:299] Negotiation successful
0929 16:47:29.587643 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260929 16:47:29.598387 23442 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:47:29.596085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34230 (local address 127.0.197.124:43025)
0929 16:47:29.596280 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:29.596287 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:29.596780 (+ 493us) server_negotiation.cc:407] Connection header received
0929 16:47:29.597037 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:29.597044 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:29.597174 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:29.597297 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:29.598146 (+ 849us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34230: BlockingRecv error: recv got EOF from 127.0.0.1:34230 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 16:47:29.610957 23366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:29.603762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44898 (local address 127.0.197.125:41099)
0929 16:47:29.603989 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:29.603996 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:29.604104 (+ 108us) server_negotiation.cc:407] Connection header received
0929 16:47:29.604399 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:29.604406 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:29.604532 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:29.604637 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:47:29.605114 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:29.606308 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:29.608686 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:29.609416 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:29.609586 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:29.610087 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:47:29.610263 (+ 176us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:47:29.610360 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:29.610535 (+ 175us) server_negotiation.cc:299] Negotiation successful
0929 16:47:29.610622 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260929 16:47:29.629323 23442 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:47:29.625468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34242 (local address 127.0.197.124:43025)
0929 16:47:29.625674 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:29.625683 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:29.627748 (+ 2065us) server_negotiation.cc:407] Connection header received
0929 16:47:29.627954 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:29.627961 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:29.628081 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:29.628193 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:29.629084 (+ 891us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34242: BlockingRecv error: recv got EOF from 127.0.0.1:34242 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 16:47:29.643277 23366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:29.635737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44900 (local address 127.0.197.125:41099)
0929 16:47:29.635945 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:29.635952 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:29.636550 (+ 598us) server_negotiation.cc:407] Connection header received
0929 16:47:29.636749 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:29.636755 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:29.636853 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:29.636965 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:47:29.637461 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:29.638681 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:29.640836 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:29.641552 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:29.641712 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:29.642237 (+ 525us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:47:29.642456 (+ 219us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:47:29.642561 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:29.642736 (+ 175us) server_negotiation.cc:299] Negotiation successful
0929 16:47:29.642851 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260929 16:47:29.659195 23442 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:47:29.656829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34246 (local address 127.0.197.124:43025)
0929 16:47:29.657018 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:29.657025 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:29.657562 (+ 537us) server_negotiation.cc:407] Connection header received
0929 16:47:29.657758 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:29.657766 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:29.657879 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:29.657991 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:29.658905 (+ 914us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34246: BlockingRecv error: recv got EOF from 127.0.0.1:34246 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 16:47:29.676898 23366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:29.669234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44904 (local address 127.0.197.125:41099)
0929 16:47:29.669498 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:29.669506 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:29.670108 (+ 602us) server_negotiation.cc:407] Connection header received
0929 16:47:29.670319 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:29.670326 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:29.670424 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:29.670547 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:47:29.671116 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:29.672353 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:29.674499 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:29.675294 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:29.675447 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:29.676005 (+ 558us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:47:29.676184 (+ 179us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:47:29.676288 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:29.676473 (+ 185us) server_negotiation.cc:299] Negotiation successful
0929 16:47:29.676578 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
W20260929 16:47:29.687943 23442 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:47:29.685181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34258 (local address 127.0.197.124:43025)
0929 16:47:29.685549 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:29.685556 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:29.686220 (+ 664us) server_negotiation.cc:407] Connection header received
0929 16:47:29.686423 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:29.686430 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:29.686542 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:29.686666 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:29.687712 (+ 1046us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34258: BlockingRecv error: recv got EOF from 127.0.0.1:34258 (error 108)
Metrics: {"server-negotiator.queue_time_us":220}
I20260929 16:47:29.707840 23366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:29.700220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44908 (local address 127.0.197.125:41099)
0929 16:47:29.700446 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:29.700453 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:29.701001 (+ 548us) server_negotiation.cc:407] Connection header received
0929 16:47:29.701226 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:29.701233 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:29.701336 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:29.701440 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:47:29.701968 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:29.703244 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:29.705380 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:29.706135 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:29.706289 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:29.706866 (+ 577us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:47:29.707079 (+ 213us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:47:29.707175 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:29.707358 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 16:47:29.707459 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260929 16:47:29.719879 23442 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:47:29.717303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34260 (local address 127.0.197.124:43025)
0929 16:47:29.717502 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:29.717522 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:29.718114 (+ 592us) server_negotiation.cc:407] Connection header received
0929 16:47:29.718330 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:29.718337 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:29.718449 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:29.718576 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:29.719605 (+ 1029us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34260: BlockingRecv error: recv got EOF from 127.0.0.1:34260 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 16:47:29.753636 23366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:29.745964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44912 (local address 127.0.197.125:41099)
0929 16:47:29.746173 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:29.746182 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:29.746786 (+ 604us) server_negotiation.cc:407] Connection header received
0929 16:47:29.746953 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:29.746959 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:29.747115 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:29.747224 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:47:29.747758 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:29.748959 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:29.751055 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:29.751729 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:29.751873 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:29.752478 (+ 605us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:47:29.752694 (+ 216us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:47:29.752787 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:29.752980 (+ 193us) server_negotiation.cc:299] Negotiation successful
0929 16:47:29.753081 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260929 16:47:29.767899 23442 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:47:29.765326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34266 (local address 127.0.197.124:43025)
0929 16:47:29.765542 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:29.765548 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:29.766145 (+ 597us) server_negotiation.cc:407] Connection header received
0929 16:47:29.766364 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:29.766371 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:29.766502 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:29.766636 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:29.767651 (+ 1015us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34266: BlockingRecv error: recv got EOF from 127.0.0.1:34266 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 16:47:29.812736 23366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:29.805000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44916 (local address 127.0.197.125:41099)
0929 16:47:29.805229 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:29.805236 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:29.805808 (+ 572us) server_negotiation.cc:407] Connection header received
0929 16:47:29.806018 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:29.806024 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:29.806130 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:29.806267 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:47:29.806746 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:29.807997 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:29.809982 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:29.810839 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:29.811270 (+ 431us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:29.811728 (+ 458us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:47:29.811939 (+ 211us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:47:29.812064 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:29.812265 (+ 201us) server_negotiation.cc:299] Negotiation successful
0929 16:47:29.812365 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260929 16:47:29.823963 23442 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:47:29.821536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34270 (local address 127.0.197.124:43025)
0929 16:47:29.821728 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:29.821734 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:29.822299 (+ 565us) server_negotiation.cc:407] Connection header received
0929 16:47:29.822473 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:29.822480 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:29.822594 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:29.822719 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:29.823698 (+ 979us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34270: BlockingRecv error: recv got EOF from 127.0.0.1:34270 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 16:47:29.898182 23366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:29.891490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44922 (local address 127.0.197.125:41099)
0929 16:47:29.891767 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:29.891774 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:29.892199 (+ 425us) server_negotiation.cc:407] Connection header received
0929 16:47:29.892375 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:29.892380 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:29.892462 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:29.892549 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:47:29.893052 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:29.893863 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:29.896009 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:29.896558 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:29.896693 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:29.897319 (+ 626us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:47:29.897476 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:47:29.897567 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:29.897746 (+ 179us) server_negotiation.cc:299] Negotiation successful
0929 16:47:29.897829 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
W20260929 16:47:29.912032 23442 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:47:29.909248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34278 (local address 127.0.197.124:43025)
0929 16:47:29.909532 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:29.909539 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:29.909970 (+ 431us) server_negotiation.cc:407] Connection header received
0929 16:47:29.910329 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:29.910337 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:29.910485 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:29.910634 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:29.911753 (+ 1119us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34278: BlockingRecv error: recv got EOF from 127.0.0.1:34278 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
I20260929 16:47:30.050235 23366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:30.044038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44932 (local address 127.0.197.125:41099)
0929 16:47:30.044351 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:30.044359 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:30.044695 (+ 336us) server_negotiation.cc:407] Connection header received
0929 16:47:30.044973 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:30.044978 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:30.045069 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:30.045193 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:47:30.045649 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:30.046536 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:30.048325 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:30.048795 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:30.048959 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:30.049432 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:47:30.049553 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:47:30.049667 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:30.049833 (+ 166us) server_negotiation.cc:299] Negotiation successful
0929 16:47:30.049927 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
W20260929 16:47:30.058554 23442 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:47:30.056536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34290 (local address 127.0.197.124:43025)
0929 16:47:30.056800 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:30.056823 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:30.057169 (+ 346us) server_negotiation.cc:407] Connection header received
0929 16:47:30.057321 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:30.057326 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:30.057412 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:30.057509 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:30.058297 (+ 788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34290: BlockingRecv error: recv got EOF from 127.0.0.1:34290 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 16:47:30.325058 23366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:30.319841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44946 (local address 127.0.197.125:41099)
0929 16:47:30.320097 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:30.320104 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:30.320253 (+ 149us) server_negotiation.cc:407] Connection header received
0929 16:47:30.320506 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:30.320512 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:30.320581 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:30.320683 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:47:30.321132 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:30.321958 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:30.323356 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:30.323771 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:30.323908 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:30.324312 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:47:30.324422 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:47:30.324522 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:30.324686 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 16:47:30.324770 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
W20260929 16:47:30.333351 23442 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:47:30.331471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34306 (local address 127.0.197.124:43025)
0929 16:47:30.331713 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:30.331719 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:30.331875 (+ 156us) server_negotiation.cc:407] Connection header received
0929 16:47:30.332123 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:30.332128 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:30.332204 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:30.332318 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:30.333168 (+ 850us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34306: BlockingRecv error: recv got EOF from 127.0.0.1:34306 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 16:47:30.341354 23481 catalog_manager.cc:1259] Loaded cluster ID: 18dddb1e795d4b19beb257e6bc420318
I20260929 16:47:30.341504 23481 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1be9b47795394e6293194b4b30b01663: loading cluster ID for follower catalog manager: success
I20260929 16:47:30.345817 23481 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1be9b47795394e6293194b4b30b01663: acquiring CA information for follower catalog manager: success
I20260929 16:47:30.348930 23481 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1be9b47795394e6293194b4b30b01663: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 16:47:30.856732 23620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:30.850302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44952 (local address 127.0.197.125:41099)
0929 16:47:30.851059 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:30.851067 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:30.851087 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:47:30.851266 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:30.851271 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:30.851359 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:30.851473 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:47:30.851919 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:30.852676 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:30.854810 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:30.855323 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:30.855488 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:30.855900 (+ 412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:47:30.856024 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:47:30.856132 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:30.856284 (+ 152us) server_negotiation.cc:299] Negotiation successful
0929 16:47:30.856374 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":457,"threads_started":1}
I20260929 16:47:30.869926 23626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:30.863509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34312 (local address 127.0.197.124:43025)
0929 16:47:30.864012 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:30.864017 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:30.864130 (+ 113us) server_negotiation.cc:407] Connection header received
0929 16:47:30.864284 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:30.864289 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:30.864466 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:30.864582 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:47:30.865099 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:30.865862 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:30.867518 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:30.867990 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:30.868106 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:30.868640 (+ 534us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:47:30.869215 (+ 575us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:47:30.869340 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:30.869481 (+ 141us) server_negotiation.cc:299] Negotiation successful
0929 16:47:30.869563 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":267,"threads_started":1}
I20260929 16:47:30.887107 23620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:30.877391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44962 (local address 127.0.197.125:41099)
0929 16:47:30.877760 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:30.877771 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:30.878790 (+ 1019us) server_negotiation.cc:407] Connection header received
0929 16:47:30.879078 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:30.879084 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:30.879175 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:30.879244 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:47:30.880215 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:30.881366 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:30.884255 (+ 2889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:30.885044 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:30.885179 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:30.886013 (+ 834us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:47:30.886178 (+ 165us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:47:30.886243 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:30.886556 (+ 313us) server_negotiation.cc:299] Negotiation successful
0929 16:47:30.886673 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198}
I20260929 16:47:30.887405 23626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:30.878996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34314 (local address 127.0.197.124:43025)
0929 16:47:30.879211 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:30.879219 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:30.879652 (+ 433us) server_negotiation.cc:407] Connection header received
0929 16:47:30.879897 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:30.879902 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:30.879993 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:30.880101 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:47:30.880641 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:30.881711 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:30.885132 (+ 3421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:30.885818 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:30.886071 (+ 253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:30.886532 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:47:30.886660 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:47:30.886743 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:30.887062 (+ 319us) server_negotiation.cc:299] Negotiation successful
0929 16:47:30.887146 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 16:47:30.888136 23633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:30.877411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51822 (local address 127.0.197.126:33615)
0929 16:47:30.878167 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:30.878175 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:30.878991 (+ 816us) server_negotiation.cc:407] Connection header received
0929 16:47:30.879139 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:30.879144 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:30.879400 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:30.879598 (+ 198us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:47:30.880366 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:30.881412 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:30.884944 (+ 3532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:30.885587 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:30.885749 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:30.886889 (+ 1140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:47:30.887403 (+ 514us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:47:30.887533 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:30.887744 (+ 211us) server_negotiation.cc:299] Negotiation successful
0929 16:47:30.887833 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":530,"threads_started":1}
I20260929 16:47:30.894922 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 23220
I20260929 16:47:30.896140 23282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:47:31.013569 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 23220
I20260929 16:47:31.035512 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 23294
I20260929 16:47:31.036655 23356 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:47:31.149531 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 23294
I20260929 16:47:31.170879 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 23369
I20260929 16:47:31.172178 23431 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:47:31.280737 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 23369
2026-09-29T16:47:31Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5194 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:47:31 dist-test-slave-l31f krb5kdc[23640](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:47:31 dist-test-slave-l31f krb5kdc[23640](info): set up 2 sockets
Sep 29 16:47:31 dist-test-slave-l31f krb5kdc[23640](info): commencing operation
krb5kdc: starting...
W20260929 16:47:32.411689 789 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/krb5kdc/test-user.keytab.
Sep 29 16:47:32 dist-test-slave-l31f krb5kdc[23640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:47:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:47:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.197.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
Entry for principal kudu/127.0.197.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab.
I20260929 16:47:32.699921 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:35461
--webserver_interface=127.0.197.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:38411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.126:35461,127.0.197.125:44579,127.0.197.124:43445
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:47:33.060750 23653 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 16:47:33.061163 23653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:47:33.061281 23653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:47:33.061424 23653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:47:33.069005 23653 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:47:33.069118 23653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:47:33.069175 23653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:47:33.069219 23653 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:47:33.069324 23653 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:47:33.079669 23653 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:38411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.197.126:35461,127.0.197.125:44579,127.0.197.124:43445
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/krb5kdc/kudu_127.0.197.126.keytab
--principal=kudu/127.0.197.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.126:35461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.197.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:47:33.081110 23653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:47:33.082500 23653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:47:33.095325 23658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:47:33.095347 23659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:47:33.095268 23661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:47:33.096334 23653 server_base.cc:1034] running on GCE node
I20260929 16:47:33.097237 23653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:47:33.098765 23653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:47:33.100020 23653 hybrid_clock.cc:648] HybridClock initialized: now 1790700453099936 us; error 66 us; skew 500 ppm
I20260929 16:47:33.100389 23653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:47:33 dist-test-slave-l31f krb5kdc[23640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:47:33.104133 23653 init.cc:377] Logged in from keytab as kudu/127.0.197.126@KRBTEST.COM (short username kudu)
I20260929 16:47:33.105818 23653 webserver.cc:460] Webserver started at http://127.0.197.126:43655/ using document root <none> and password file <none>
I20260929 16:47:33.106611 23653 fs_manager.cc:359] Metadata directory not provided
I20260929 16:47:33.106763 23653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:47:33.107203 23653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:47:33.110445 23653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/minicluster-data/master-0/data/instance:
uuid: "0d87403c02d94935bdaa198173c21cc1"
format_stamp: "Formatted at 2026-09-29 16:47:33 on dist-test-slave-l31f"
server_key: "7bd7338fd6472ded86674a6fc9da24a7"
server_key_iv: "2e5400abd2336804ae2f31b9ea7d439b"
server_key_version: "encryptionkey@0"
I20260929 16:47:33.111392 23653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/minicluster-data/master-0/wal/instance:
uuid: "0d87403c02d94935bdaa198173c21cc1"
format_stamp: "Formatted at 2026-09-29 16:47:33 on dist-test-slave-l31f"
server_key: "7bd7338fd6472ded86674a6fc9da24a7"
server_key_iv: "2e5400abd2336804ae2f31b9ea7d439b"
server_key_version: "encryptionkey@0"
I20260929 16:47:33.117443 23653 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.005s
I20260929 16:47:33.122010 23668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:47:33.123742 23653 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260929 16:47:33.123999 23653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/minicluster-data/master-0/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/minicluster-data/master-0/wal
uuid: "0d87403c02d94935bdaa198173c21cc1"
format_stamp: "Formatted at 2026-09-29 16:47:33 on dist-test-slave-l31f"
server_key: "7bd7338fd6472ded86674a6fc9da24a7"
server_key_iv: "2e5400abd2336804ae2f31b9ea7d439b"
server_key_version: "encryptionkey@0"
I20260929 16:47:33.124279 23653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:47:33.141935 23653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:47:33.147197 23653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:47:33.147527 23653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:47:33.169469 23653 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.126:35461
I20260929 16:47:33.169482 23719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.126:35461 every 8 connection(s)
I20260929 16:47:33.171051 23653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/minicluster-data/master-0/data/info.pb
I20260929 16:47:33.175570 23720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:47:33.179559 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 23653
I20260929 16:47:33.179780 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/minicluster-data/master-0/wal/instance
I20260929 16:47:33.180325 789 external_mini_cluster.cc:1428] Setting key 51fd19a5fc6d07c7ac4d6045e3f00e8d
I20260929 16:47:33.185437 23720 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.197.126" port: 35461 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:47:33 dist-test-slave-l31f krb5kdc[23640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.197.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.125@KRBTEST.COM" created.
I20260929 16:47:33.218828 23722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:33.190789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41614 (local address 127.0.197.126:35461)
0929 16:47:33.191842 (+ 1053us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:33.191869 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:33.192440 (+ 571us) server_negotiation.cc:407] Connection header received
0929 16:47:33.196019 (+ 3579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:33.196027 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:33.196344 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:33.196549 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:33.198674 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:33.200510 (+ 1836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:33.203872 (+ 3362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:33.204652 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:33.204873 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:33.209818 (+ 4945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:47:33.209842 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:47:33.209857 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:47:33.209891 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:47:33.213192 (+ 3301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:33.214082 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:33.214090 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:33.214092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:33.214341 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:33.215568 (+ 1227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:33.215576 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:33.215582 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:33.216359 (+ 777us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:47:33.216554 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:33.217345 (+ 791us) server_negotiation.cc:299] Negotiation successful
0929 16:47:33.217680 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":738,"thread_start_us":431,"threads_started":1}
I20260929 16:47:33.218873 23723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:33.190788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:35461 (local address 127.0.0.1:41614)
0929 16:47:33.192197 (+ 1409us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:33.192252 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 16:47:33.195418 (+ 3166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:47:33.196667 (+ 1249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:47:33.196680 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:47:33.197609 (+ 929us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:47:33.198150 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:47:33.198183 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:47:33.200827 (+ 2644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:47:33.200837 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:47:33.203464 (+ 2627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:47:33.203506 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:47:33.205197 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:47:33.205220 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:47:33.205519 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:33.206345 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:47:33.206385 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:47:33.209515 (+ 3130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:47:33.213360 (+ 3845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:47:33.213374 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:47:33.213388 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:47:33.213872 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:47:33.214752 (+ 880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:47:33.214758 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:47:33.214762 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:47:33.215280 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:47:33.216566 (+ 1286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:47:33.216578 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:47:33.216881 (+ 303us) client_negotiation.cc:769] Sending connection context
0929 16:47:33.217209 (+ 328us) client_negotiation.cc:240] Negotiation successful
0929 16:47:33.217917 (+ 708us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1083,"mutex_wait_us":126,"thread_start_us":821,"threads_started":1}
I20260929 16:47:33.222761 23720 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.197.125" port: 44579 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:47:33.224277 23723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:33.223578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.125:44579 (local address 127.0.0.1:34412)
0929 16:47:33.223877 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:33.224095 (+ 218us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.125:44579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260929 16:47:33.224712 23672 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.197.125:44579: connect: Connection refused (error 111)
I20260929 16:47:33.226613 23723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:33.226061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.125:44579 (local address 127.0.0.1:34422)
0929 16:47:33.226297 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:33.226410 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.125:44579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260929 16:47:33.226850 23720 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.197.125:44579: Network error: Client connection negotiation failed: client connection to 127.0.197.125:44579: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/krb5kdc/kudu_127.0.197.125.keytab.
Entry for principal kudu/127.0.197.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/krb5kdc/kudu_127.0.197.125.keytab.
I20260929 16:47:33.254706 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.125:44579
--webserver_interface=127.0.197.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/krb5kdc/kudu_127.0.197.125.keytab
--principal=kudu/127.0.197.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:38411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.126:35461,127.0.197.125:44579,127.0.197.124:43445
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:47:33.276319 23720 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.197.125" port: 44579 } attempt: 1
I20260929 16:47:33.277488 23723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:33.277037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.125:44579 (local address 127.0.0.1:34430)
0929 16:47:33.277241 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:33.277352 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.125:44579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 16:47:33.278674 23723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:33.278261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.125:44579 (local address 127.0.0.1:34446)
0929 16:47:33.278467 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:33.278535 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.125:44579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260929 16:47:33.278863 23720 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.197.125:44579: Network error: Client connection negotiation failed: client connection to 127.0.197.125:44579: connect: Connection refused (error 111)
I20260929 16:47:33.347178 23720 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.197.125" port: 44579 } attempt: 2
I20260929 16:47:33.348426 23723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:33.347919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.125:44579 (local address 127.0.0.1:34452)
0929 16:47:33.348190 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:33.348277 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.125:44579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260929 16:47:33.349604 23723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:33.349183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.125:44579 (local address 127.0.0.1:34468)
0929 16:47:33.349354 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:33.349454 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.125:44579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260929 16:47:33.349802 23720 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.197.125:44579: Network error: Client connection negotiation failed: client connection to 127.0.197.125:44579: connect: Connection refused (error 111)
I20260929 16:47:33.441097 23720 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.197.125" port: 44579 } attempt: 3
I20260929 16:47:33.442296 23723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:33.441854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.125:44579 (local address 127.0.0.1:34484)
0929 16:47:33.442068 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:33.442165 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.125:44579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260929 16:47:33.443426 23723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:33.443082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.125:44579 (local address 127.0.0.1:34486)
0929 16:47:33.443227 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:33.443289 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.125:44579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 16:47:33.443607 23720 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.197.125:44579: Network error: Client connection negotiation failed: client connection to 127.0.197.125:44579: connect: Connection refused (error 111)
I20260929 16:47:33.586874 23720 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.197.125" port: 44579 } attempt: 4
I20260929 16:47:33.588238 23723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:33.587666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.125:44579 (local address 127.0.0.1:34488)
0929 16:47:33.587996 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:33.588090 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.125:44579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
I20260929 16:47:33.589352 23723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:33.588964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.125:44579 (local address 127.0.0.1:34496)
0929 16:47:33.589125 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:33.589219 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.125:44579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260929 16:47:33.589560 23720 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.197.125:44579: Network error: Client connection negotiation failed: client connection to 127.0.197.125:44579: connect: Connection refused (error 111)
W20260929 16:47:33.605779 23727 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 16:47:33.606240 23727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:47:33.606384 23727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:47:33.606532 23727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:47:33.614184 23727 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:47:33.614357 23727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:47:33.614473 23727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:47:33.614568 23727 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:47:33.614658 23727 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:47:33.625245 23727 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:38411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.197.126:35461,127.0.197.125:44579,127.0.197.124:43445
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/krb5kdc/kudu_127.0.197.125.keytab
--principal=kudu/127.0.197.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.125:44579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.197.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:47:33.626698 23727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:47:33.628275 23727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:47:33.642413 23732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:47:33.642372 23735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:47:33.642482 23733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:47:33.643610 23727 server_base.cc:1034] running on GCE node
I20260929 16:47:33.644618 23727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:47:33.646289 23727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:47:33.647596 23727 hybrid_clock.cc:648] HybridClock initialized: now 1790700453647515 us; error 56 us; skew 500 ppm
I20260929 16:47:33.647986 23727 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:47:33 dist-test-slave-l31f krb5kdc[23640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:47:33.651754 23727 init.cc:377] Logged in from keytab as kudu/127.0.197.125@KRBTEST.COM (short username kudu)
I20260929 16:47:33.653486 23727 webserver.cc:460] Webserver started at http://127.0.197.125:42159/ using document root <none> and password file <none>
I20260929 16:47:33.654291 23727 fs_manager.cc:359] Metadata directory not provided
I20260929 16:47:33.654452 23727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:47:33.654805 23727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:47:33.658234 23727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/minicluster-data/master-1/data/instance:
uuid: "f980ff460ed643cab40f4a477ac52c43"
format_stamp: "Formatted at 2026-09-29 16:47:33 on dist-test-slave-l31f"
server_key: "e7f1922e12b840bbb102fb78951178f9"
server_key_iv: "38ffca9124f4ff5335ca1352f7b897c8"
server_key_version: "encryptionkey@0"
I20260929 16:47:33.659279 23727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/minicluster-data/master-1/wal/instance:
uuid: "f980ff460ed643cab40f4a477ac52c43"
format_stamp: "Formatted at 2026-09-29 16:47:33 on dist-test-slave-l31f"
server_key: "e7f1922e12b840bbb102fb78951178f9"
server_key_iv: "38ffca9124f4ff5335ca1352f7b897c8"
server_key_version: "encryptionkey@0"
I20260929 16:47:33.665400 23727 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 16:47:33.669649 23742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:47:33.671829 23727 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:47:33.672086 23727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/minicluster-data/master-1/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/minicluster-data/master-1/wal
uuid: "f980ff460ed643cab40f4a477ac52c43"
format_stamp: "Formatted at 2026-09-29 16:47:33 on dist-test-slave-l31f"
server_key: "e7f1922e12b840bbb102fb78951178f9"
server_key_iv: "38ffca9124f4ff5335ca1352f7b897c8"
server_key_version: "encryptionkey@0"
I20260929 16:47:33.672379 23727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:47:33.690892 23727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:47:33.695763 23727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:47:33.696105 23727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:47:33.716265 23727 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.125:44579
I20260929 16:47:33.716318 23793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.125:44579 every 8 connection(s)
I20260929 16:47:33.717756 23727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/minicluster-data/master-1/data/info.pb
I20260929 16:47:33.722075 23794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:47:33.723476 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 23727
I20260929 16:47:33.723673 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/minicluster-data/master-1/wal/instance
I20260929 16:47:33.724108 789 external_mini_cluster.cc:1428] Setting key cddbb80438926a919b28d152bf3b52d3
I20260929 16:47:33.730096 23794 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.197.126" port: 35461 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:47:33 dist-test-slave-l31f krb5kdc[23640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.125@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.197.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.197.124@KRBTEST.COM" created.
I20260929 16:47:33.754119 23797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:33.733047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41626 (local address 127.0.197.126:35461)
0929 16:47:33.733596 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:33.733601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:33.734814 (+ 1213us) server_negotiation.cc:407] Connection header received
0929 16:47:33.737105 (+ 2291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:33.737113 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:33.737217 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:33.737352 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:33.739239 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:33.740062 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:33.741752 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:33.742210 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:33.742357 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:33.746746 (+ 4389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:47:33.746752 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:47:33.746755 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:47:33.746802 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:47:33.750157 (+ 3355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:33.751144 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:33.751149 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:33.751151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:33.751311 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:33.751805 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:33.751811 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:33.751814 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:33.752190 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:47:33.752359 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:33.753274 (+ 915us) server_negotiation.cc:299] Negotiation successful
0929 16:47:33.753566 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":266,"threads_started":1}
I20260929 16:47:33.754772 23798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:33.733752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:35461 (local address 127.0.0.1:41626)
0929 16:47:33.734617 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:33.734669 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 16:47:33.736756 (+ 2087us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:47:33.737645 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:47:33.737655 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:47:33.738275 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:47:33.738928 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:47:33.738950 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:47:33.740267 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:47:33.740271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:47:33.741574 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:47:33.741590 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:47:33.742494 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:47:33.742503 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:47:33.742746 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:33.743593 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:47:33.743619 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:47:33.746424 (+ 2805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:47:33.750365 (+ 3941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:47:33.750379 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:47:33.750403 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:47:33.750942 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:47:33.751460 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:47:33.751465 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:47:33.751469 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:47:33.751673 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:47:33.752393 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:47:33.752406 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:47:33.752852 (+ 446us) client_negotiation.cc:769] Sending connection context
0929 16:47:33.753118 (+ 266us) client_negotiation.cc:240] Negotiation successful
0929 16:47:33.753565 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":352,"threads_started":1}
I20260929 16:47:33.756755 23794 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.197.125" port: 44579 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:47:33 dist-test-slave-l31f krb5kdc[23640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.125@KRBTEST.COM for kudu/127.0.197.125@KRBTEST.COM
I20260929 16:47:33.772899 23799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:33.757749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34504 (local address 127.0.197.125:44579)
0929 16:47:33.758382 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:33.758388 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:33.758415 (+ 27us) server_negotiation.cc:407] Connection header received
0929 16:47:33.758465 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:33.758471 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:33.758657 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:33.758835 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:33.759657 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:33.760734 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:33.761934 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:33.762501 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:33.762644 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:33.766370 (+ 3726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:47:33.766386 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:47:33.766402 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:47:33.766436 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:47:33.769465 (+ 3029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:33.770428 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:33.770436 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:33.770438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:33.770564 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:33.771130 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:33.771133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:33.771135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:33.771559 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:47:33.771725 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:33.772203 (+ 478us) server_negotiation.cc:299] Negotiation successful
0929 16:47:33.772564 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":184,"server-negotiator.queue_time_us":489,"thread_start_us":354,"threads_started":1}
I20260929 16:47:33.773006 23798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:33.757423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.125:44579 (local address 127.0.0.1:34504)
0929 16:47:33.757651 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:33.757667 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:47:33.757766 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:47:33.758819 (+ 1053us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:47:33.758823 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:47:33.759330 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:47:33.759471 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:47:33.759480 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:47:33.760897 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:47:33.760904 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:47:33.761782 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:47:33.761794 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:47:33.762657 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:47:33.762662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:47:33.762900 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:33.763645 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:47:33.763663 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:47:33.766102 (+ 2439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:47:33.769646 (+ 3544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:47:33.769655 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:47:33.769662 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:47:33.770257 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:47:33.770711 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:47:33.770718 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:47:33.770724 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:47:33.770895 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:47:33.771703 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:47:33.771711 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:47:33.771884 (+ 173us) client_negotiation.cc:769] Sending connection context
0929 16:47:33.772053 (+ 169us) client_negotiation.cc:240] Negotiation successful
0929 16:47:33.772440 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"mutex_wait_us":4}
I20260929 16:47:33.775760 23794 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43445 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:47:33.777226 23798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:33.776564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43445 (local address 127.0.0.1:58514)
0929 16:47:33.776770 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:33.776992 (+ 222us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260929 16:47:33.777592 23743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43445: connect: Connection refused (error 111)
I20260929 16:47:33.779358 23798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:33.778905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43445 (local address 127.0.0.1:58516)
0929 16:47:33.779145 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:33.779242 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260929 16:47:33.779608 23794 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.197.124:43445: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43445: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.197.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/krb5kdc/kudu_127.0.197.124.keytab.
Entry for principal kudu/127.0.197.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/krb5kdc/kudu_127.0.197.124.keytab.
I20260929 16:47:33.788666 789 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
/tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.124:43445
--webserver_interface=127.0.197.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/krb5kdc/kudu_127.0.197.124.keytab
--principal=kudu/127.0.197.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.197.84:38411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.197.126:35461,127.0.197.125:44579,127.0.197.124:43445
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:47:33.828925 23794 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43445 } attempt: 1
I20260929 16:47:33.830351 23798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:33.829739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43445 (local address 127.0.0.1:58528)
0929 16:47:33.829984 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:33.830130 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260929 16:47:33.831789 23798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:33.831362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43445 (local address 127.0.0.1:58538)
0929 16:47:33.831556 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:33.831666 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260929 16:47:33.832034 23794 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.197.124:43445: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43445: connect: Connection refused (error 111)
I20260929 16:47:33.888882 23720 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.197.125" port: 44579 } attempt: 5
Sep 29 16:47:33 dist-test-slave-l31f krb5kdc[23640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for kudu/127.0.197.125@KRBTEST.COM
I20260929 16:47:33.900262 23794 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43445 } attempt: 2
I20260929 16:47:33.901364 23798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:33.900930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43445 (local address 127.0.0.1:58544)
0929 16:47:33.901115 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:33.901204 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 16:47:33.904544 23798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:33.902099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43445 (local address 127.0.0.1:58552)
0929 16:47:33.904125 (+ 2026us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:33.904251 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1884,"threads_started":1}
W20260929 16:47:33.905644 23794 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.197.124:43445: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43445: connect: Connection refused (error 111)
I20260929 16:47:33.907088 23799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:33.889903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34506 (local address 127.0.197.125:44579)
0929 16:47:33.890103 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:33.890110 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:33.890133 (+ 23us) server_negotiation.cc:407] Connection header received
0929 16:47:33.890350 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:33.890354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:33.890462 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:33.890592 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:33.891503 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:33.892232 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:33.893872 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:33.894698 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:33.894853 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:33.899083 (+ 4230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:47:33.899097 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:47:33.899102 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:47:33.899140 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:47:33.903720 (+ 4580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:33.904843 (+ 1123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:33.904849 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:33.904853 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:33.905015 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:33.905550 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:33.905556 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:33.905559 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:33.905953 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:47:33.906072 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:33.906574 (+ 502us) server_negotiation.cc:299] Negotiation successful
0929 16:47:33.906750 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 16:47:33.907176 23723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:33.889753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.125:44579 (local address 127.0.0.1:34506)
0929 16:47:33.889991 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:33.890014 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 16:47:33.890153 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:47:33.890621 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:47:33.890625 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:47:33.891151 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:47:33.891335 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:47:33.891345 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:47:33.892375 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:47:33.892382 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:47:33.893684 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:47:33.893703 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:47:33.895077 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:47:33.895086 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:47:33.895340 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:33.896081 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:47:33.896109 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:47:33.898769 (+ 2660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:47:33.903967 (+ 5198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:47:33.903974 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:47:33.903978 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:47:33.904572 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:47:33.905177 (+ 605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:47:33.905185 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:47:33.905191 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:47:33.905397 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:47:33.906076 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:47:33.906081 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:47:33.906268 (+ 187us) client_negotiation.cc:769] Sending connection context
0929 16:47:33.906454 (+ 186us) client_negotiation.cc:240] Negotiation successful
0929 16:47:33.906796 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 16:47:33.908326 23720 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43445 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:47:33.909510 23723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:33.908965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43445 (local address 127.0.0.1:58558)
0929 16:47:33.909194 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:33.909332 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 16:47:33.910694 23723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:33.910237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43445 (local address 127.0.0.1:58572)
0929 16:47:33.910439 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:33.910560 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260929 16:47:33.910957 23720 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.197.124:43445: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43445: connect: Connection refused (error 111)
I20260929 16:47:33.962284 23720 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43445 } attempt: 1
I20260929 16:47:33.963725 23723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:33.963167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43445 (local address 127.0.0.1:58580)
0929 16:47:33.963389 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:33.963504 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 16:47:33.964936 23723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:33.964513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43445 (local address 127.0.0.1:58596)
0929 16:47:33.964691 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:33.964785 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260929 16:47:33.965227 23720 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.197.124:43445: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43445: connect: Connection refused (error 111)
I20260929 16:47:33.996932 23794 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43445 } attempt: 3
I20260929 16:47:33.998430 23803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:33.997668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43445 (local address 127.0.0.1:58602)
0929 16:47:33.997919 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:33.998174 (+ 255us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260929 16:47:33.999889 23803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:33.999304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43445 (local address 127.0.0.1:58618)
0929 16:47:33.999591 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:33.999705 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
W20260929 16:47:34.000188 23794 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.197.124:43445: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43445: connect: Connection refused (error 111)
I20260929 16:47:34.033520 23720 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43445 } attempt: 2
I20260929 16:47:34.034940 23723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:34.034249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43445 (local address 127.0.0.1:58620)
0929 16:47:34.034553 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:34.034685 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260929 16:47:34.036294 23723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:34.035824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43445 (local address 127.0.0.1:58626)
0929 16:47:34.036025 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:34.036126 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260929 16:47:34.036572 23720 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.197.124:43445: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43445: connect: Connection refused (error 111)
I20260929 16:47:34.142839 23720 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43445 } attempt: 3
I20260929 16:47:34.143437 23794 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43445 } attempt: 4
I20260929 16:47:34.144214 23723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:34.143615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43445 (local address 127.0.0.1:58642)
0929 16:47:34.143920 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:34.144035 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260929 16:47:34.144716 23803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:34.144192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43445 (local address 127.0.0.1:58654)
0929 16:47:34.144416 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:34.144517 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260929 16:47:34.145723 23723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:34.145262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43445 (local address 127.0.0.1:58656)
0929 16:47:34.145488 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:34.145575 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260929 16:47:34.145843 23803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:34.145392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43445 (local address 127.0.0.1:58666)
0929 16:47:34.145595 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:34.145668 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
W20260929 16:47:34.146018 23720 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.197.124:43445: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43445: connect: Connection refused (error 111)
W20260929 16:47:34.146128 23794 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.197.124:43445: Network error: Client connection negotiation failed: client connection to 127.0.197.124:43445: connect: Connection refused (error 111)
W20260929 16:47:34.150100 23802 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 16:47:34.150496 23802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:47:34.150600 23802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:47:34.150741 23802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:47:34.158564 23802 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:47:34.158679 23802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:47:34.158783 23802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:47:34.158864 23802 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:47:34.158954 23802 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:47:34.169867 23802 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.197.84:38411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.197.126:35461,127.0.197.125:44579,127.0.197.124:43445
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/krb5kdc/kudu_127.0.197.124.keytab
--principal=kudu/127.0.197.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.197.124:43445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.197.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 15:44:16 UTC on ccycloud.quasar-keyily.root.comops.site
build id 4680678
ASAN enabled
I20260929 16:47:34.171389 23802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:47:34.173393 23802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:47:34.186368 23808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:47:34.186391 23809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:47:34.186540 23811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:47:34.187997 23802 server_base.cc:1034] running on GCE node
I20260929 16:47:34.188961 23802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:47:34.190601 23802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:47:34.191900 23802 hybrid_clock.cc:648] HybridClock initialized: now 1790700454191831 us; error 45 us; skew 500 ppm
I20260929 16:47:34.192269 23802 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:47:34 dist-test-slave-l31f krb5kdc[23640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:47:34.202426 23802 init.cc:377] Logged in from keytab as kudu/127.0.197.124@KRBTEST.COM (short username kudu)
I20260929 16:47:34.204296 23802 webserver.cc:460] Webserver started at http://127.0.197.124:43813/ using document root <none> and password file <none>
I20260929 16:47:34.205090 23802 fs_manager.cc:359] Metadata directory not provided
I20260929 16:47:34.205240 23802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:47:34.205598 23802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:47:34.208923 23802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/minicluster-data/master-2/data/instance:
uuid: "45f02b7b782346e58158981174a8c839"
format_stamp: "Formatted at 2026-09-29 16:47:34 on dist-test-slave-l31f"
server_key: "fa013aa7e727ff2f190b1e7ad62d7a87"
server_key_iv: "08abaca2b7a533ca960e047fc15f016b"
server_key_version: "encryptionkey@0"
I20260929 16:47:34.209834 23802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/minicluster-data/master-2/wal/instance:
uuid: "45f02b7b782346e58158981174a8c839"
format_stamp: "Formatted at 2026-09-29 16:47:34 on dist-test-slave-l31f"
server_key: "fa013aa7e727ff2f190b1e7ad62d7a87"
server_key_iv: "08abaca2b7a533ca960e047fc15f016b"
server_key_version: "encryptionkey@0"
I20260929 16:47:34.216171 23802 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260929 16:47:34.220260 23818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:47:34.221791 23802 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:47:34.222041 23802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/minicluster-data/master-2/data,/tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/minicluster-data/master-2/wal
uuid: "45f02b7b782346e58158981174a8c839"
format_stamp: "Formatted at 2026-09-29 16:47:34 on dist-test-slave-l31f"
server_key: "fa013aa7e727ff2f190b1e7ad62d7a87"
server_key_iv: "08abaca2b7a533ca960e047fc15f016b"
server_key_version: "encryptionkey@0"
I20260929 16:47:34.222357 23802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:47:34.251106 23802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:47:34.256089 23802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:47:34.256397 23802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:47:34.276952 23802 rpc_server.cc:307] RPC server started. Bound to: 127.0.197.124:43445
I20260929 16:47:34.276975 23869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.197.124:43445 every 8 connection(s)
I20260929 16:47:34.278718 23802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/minicluster-data/master-2/data/info.pb
I20260929 16:47:34.282734 23870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:47:34.287117 789 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu as pid 23802
I20260929 16:47:34.287336 789 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQmsv3Z/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790700068255611-789-0/minicluster-data/master-2/wal/instance
I20260929 16:47:34.287771 789 external_mini_cluster.cc:1428] Setting key d02b108dcd0dd50533213450fc0750ad
I20260929 16:47:34.290417 23870 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.197.126" port: 35461 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:47:34 dist-test-slave-l31f krb5kdc[23640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.124@KRBTEST.COM
Sep 29 16:47:34 dist-test-slave-l31f krb5kdc[23640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.124@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
I20260929 16:47:34.313313 23874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:34.294940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44634 (local address 127.0.197.126:35461)
0929 16:47:34.295596 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:34.295602 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:34.296319 (+ 717us) server_negotiation.cc:407] Connection header received
0929 16:47:34.296518 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:34.296524 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:34.296618 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:34.296740 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:34.298240 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:34.299106 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:34.300877 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:34.301340 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:34.301453 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:34.305791 (+ 4338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:47:34.305798 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:47:34.305802 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:47:34.305841 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:47:34.309589 (+ 3748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:34.310645 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:34.310651 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:34.310655 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:34.310832 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:34.311331 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:34.311334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:34.311336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:34.311656 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:47:34.311789 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:34.312548 (+ 759us) server_negotiation.cc:299] Negotiation successful
0929 16:47:34.312796 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":367,"threads_started":1}
I20260929 16:47:34.314239 23875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:34.294954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.126:35461 (local address 127.0.0.1:44634)
0929 16:47:34.296149 (+ 1195us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:34.296199 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 16:47:34.296377 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:47:34.296896 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:47:34.296902 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:47:34.297718 (+ 816us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:47:34.298063 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:47:34.298078 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:47:34.299259 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:47:34.299263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:47:34.300714 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:47:34.300727 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:47:34.301608 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:47:34.301617 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:47:34.301846 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:34.302535 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:47:34.302572 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:47:34.305484 (+ 2912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:47:34.309807 (+ 4323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:47:34.309820 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:47:34.309832 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:47:34.310519 (+ 687us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:47:34.310994 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:47:34.311001 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:47:34.311031 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:47:34.311203 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:47:34.311801 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:47:34.311811 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:47:34.312088 (+ 277us) client_negotiation.cc:769] Sending connection context
0929 16:47:34.312397 (+ 309us) client_negotiation.cc:240] Negotiation successful
0929 16:47:34.313134 (+ 737us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":937,"mutex_wait_us":228,"spinlock_wait_cycles":77056,"thread_start_us":724,"threads_started":1}
I20260929 16:47:34.314254 23873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:34.289755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41078 (local address 127.0.197.124:43445)
0929 16:47:34.290636 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:34.290649 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:34.290694 (+ 45us) server_negotiation.cc:407] Connection header received
0929 16:47:34.292636 (+ 1942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:34.292664 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:34.293182 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:34.293552 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:34.295461 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:34.297342 (+ 1881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:34.298957 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:34.299914 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:34.300063 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:34.305263 (+ 5200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:47:34.305278 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:47:34.305301 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:47:34.305347 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:47:34.308978 (+ 3631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:34.310058 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:34.310068 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:34.310071 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:34.310277 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:34.310896 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:34.310902 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:34.310905 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:34.311804 (+ 899us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:47:34.312133 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:34.312421 (+ 288us) server_negotiation.cc:299] Negotiation successful
0929 16:47:34.312938 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":78,"server-negotiator.queue_time_us":641,"thread_start_us":421,"threads_started":1}
I20260929 16:47:34.316819 23870 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.197.125" port: 44579 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:47:34.323262 23720 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43445 } attempt: 4
Sep 29 16:47:34 dist-test-slave-l31f krb5kdc[23640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.124@KRBTEST.COM for kudu/127.0.197.125@KRBTEST.COM
Sep 29 16:47:34 dist-test-slave-l31f krb5kdc[23640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.126@KRBTEST.COM
Sep 29 16:47:34 dist-test-slave-l31f krb5kdc[23640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.126@KRBTEST.COM for kudu/127.0.197.124@KRBTEST.COM
I20260929 16:47:34.332883 23875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:34.317632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.125:44579 (local address 127.0.0.1:50988)
0929 16:47:34.317816 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:34.317842 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 16:47:34.317983 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:47:34.318362 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:47:34.318367 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:47:34.318760 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:47:34.318906 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:47:34.318913 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:47:34.319950 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:47:34.319955 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:47:34.320849 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:47:34.320861 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:47:34.321635 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:47:34.321641 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:47:34.321857 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:34.322522 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:47:34.322541 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:47:34.325137 (+ 2596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:47:34.329596 (+ 4459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:47:34.329602 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:47:34.329608 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:47:34.330194 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:47:34.330686 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:47:34.330692 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:47:34.330696 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:47:34.330878 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:47:34.331750 (+ 872us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:47:34.331757 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:47:34.331957 (+ 200us) client_negotiation.cc:769] Sending connection context
0929 16:47:34.332129 (+ 172us) client_negotiation.cc:240] Negotiation successful
0929 16:47:34.332465 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 16:47:34.333873 23799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:34.317858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50988 (local address 127.0.197.125:44579)
0929 16:47:34.318072 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:34.318081 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:34.318098 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:47:34.318158 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:34.318162 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:34.318220 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:34.318327 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:34.319097 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:34.319796 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:34.321022 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:34.321479 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:34.321621 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:34.325378 (+ 3757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:47:34.325385 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:47:34.325391 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:47:34.325442 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:47:34.329424 (+ 3982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:34.330369 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:34.330376 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:34.330380 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:34.330556 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:34.331074 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:34.331080 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:34.331083 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:34.331502 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:47:34.331637 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:34.333245 (+ 1608us) server_negotiation.cc:299] Negotiation successful
0929 16:47:34.333503 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 16:47:34.335101 23870 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43445 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:47:34.336755 23874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:34.318290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44646 (local address 127.0.197.126:35461)
0929 16:47:34.318491 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:34.318499 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:34.318519 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:47:34.318594 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:34.318598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:34.318653 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:34.318766 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:34.320480 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:34.321186 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:34.322621 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:34.323165 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:34.323342 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:34.329190 (+ 5848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:47:34.329198 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:47:34.329204 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:47:34.329255 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:47:34.333141 (+ 3886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:34.334098 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:34.334103 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:34.334107 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:34.334290 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:34.334773 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:34.334778 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:34.334782 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:34.335237 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:47:34.335533 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:34.336117 (+ 584us) server_negotiation.cc:299] Negotiation successful
0929 16:47:34.336372 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 16:47:34.343627 23723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:34.323979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43445 (local address 127.0.0.1:41094)
0929 16:47:34.324210 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:34.324230 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:47:34.324325 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:47:34.325382 (+ 1057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:47:34.325387 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:47:34.325772 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:47:34.325924 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:47:34.325942 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:47:34.327327 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:47:34.327333 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:47:34.328748 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:47:34.328763 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:47:34.329558 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:47:34.329564 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:47:34.329797 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:34.330748 (+ 951us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:47:34.330765 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:47:34.333380 (+ 2615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:47:34.340616 (+ 7236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:47:34.340624 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:47:34.340630 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:47:34.341202 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:47:34.341654 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:47:34.341660 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:47:34.341664 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:47:34.341841 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:47:34.342533 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:47:34.342541 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:47:34.342720 (+ 179us) client_negotiation.cc:769] Sending connection context
0929 16:47:34.342902 (+ 182us) client_negotiation.cc:240] Negotiation successful
0929 16:47:34.343236 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260929 16:47:34.343642 23873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:34.324817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41094 (local address 127.0.197.124:43445)
0929 16:47:34.325024 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:34.325032 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:34.325052 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:47:34.325106 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:34.325112 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:34.325183 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:34.325357 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:34.326126 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:34.327177 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:34.328898 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:34.329379 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:34.329588 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:34.333586 (+ 3998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:47:34.333592 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:47:34.333595 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:47:34.333624 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:47:34.340462 (+ 6838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:34.341354 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:34.341365 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:34.341370 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:34.341533 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:34.341985 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:34.341991 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:34.341994 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:34.342371 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:47:34.342545 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:34.343027 (+ 482us) server_negotiation.cc:299] Negotiation successful
0929 16:47:34.343283 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
Sep 29 16:47:34 dist-test-slave-l31f krb5kdc[23640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.124@KRBTEST.COM for kudu/127.0.197.124@KRBTEST.COM
I20260929 16:47:34.349937 23720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d87403c02d94935bdaa198173c21cc1: Bootstrap starting.
Sep 29 16:47:34 dist-test-slave-l31f krb5kdc[23640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.197.125@KRBTEST.COM
I20260929 16:47:34.354530 23720 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d87403c02d94935bdaa198173c21cc1: Neither blocks nor log segments found. Creating new log.
I20260929 16:47:34.356037 23720 log.cc:824] T 00000000000000000000000000000000 P 0d87403c02d94935bdaa198173c21cc1: Log is configured to *not* fsync() on all Append() calls
I20260929 16:47:34.357257 23876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:34.336105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41108 (local address 127.0.197.124:43445)
0929 16:47:34.341019 (+ 4914us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:34.341025 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:34.341046 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:47:34.341099 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:34.341105 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:34.341182 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:34.341291 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:34.342352 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:34.343921 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:34.345498 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:34.346129 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:34.346249 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:34.350242 (+ 3993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:47:34.350249 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:47:34.350253 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:47:34.350297 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:47:34.354033 (+ 3736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:34.354825 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:34.354833 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:34.354837 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:34.355059 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:34.355521 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:34.355528 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:34.355534 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:34.355888 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:47:34.356056 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:34.356569 (+ 513us) server_negotiation.cc:299] Negotiation successful
0929 16:47:34.356795 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4777,"thread_start_us":4607,"threads_started":1}
I20260929 16:47:34.357298 23875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:34.335916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43445 (local address 127.0.0.1:41108)
0929 16:47:34.336239 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:34.336259 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:47:34.336364 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:47:34.341551 (+ 5187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:47:34.341554 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:47:34.342045 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:47:34.342201 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:47:34.342209 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:47:34.344080 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:47:34.344084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:47:34.345351 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:47:34.345363 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:47:34.346272 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:47:34.346278 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:47:34.346501 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:34.347050 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:47:34.347077 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:47:34.349963 (+ 2886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:47:34.354226 (+ 4263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:47:34.354232 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:47:34.354237 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:47:34.354689 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:47:34.355207 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:47:34.355213 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:47:34.355216 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:47:34.355387 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:47:34.356123 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:47:34.356127 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:47:34.356302 (+ 175us) client_negotiation.cc:769] Sending connection context
0929 16:47:34.356463 (+ 161us) client_negotiation.cc:240] Negotiation successful
0929 16:47:34.356953 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":155}
I20260929 16:47:34.360683 23799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:34.339106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51000 (local address 127.0.197.125:44579)
0929 16:47:34.339296 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:34.339302 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:34.339321 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:47:34.339386 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:34.339393 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:34.339478 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:34.339587 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:34.341680 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:34.342704 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:34.344708 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:34.345278 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:34.345405 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:34.351084 (+ 5679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:47:34.351091 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:47:34.351097 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:47:34.351141 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:47:34.354052 (+ 2911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:34.355651 (+ 1599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:34.355660 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:34.355665 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:34.355856 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:34.356313 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:34.356317 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:34.356320 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:34.356730 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:47:34.357103 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:34.359787 (+ 2684us) server_negotiation.cc:299] Negotiation successful
0929 16:47:34.360171 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 16:47:34.363221 23870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45f02b7b782346e58158981174a8c839: Bootstrap starting.
I20260929 16:47:34.366533 789 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:47:34.367295 23720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d87403c02d94935bdaa198173c21cc1: No bootstrap required, opened a new log
I20260929 16:47:34.367470 23870 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45f02b7b782346e58158981174a8c839: Neither blocks nor log segments found. Creating new log.
I20260929 16:47:34.368906 23870 log.cc:824] T 00000000000000000000000000000000 P 45f02b7b782346e58158981174a8c839: Log is configured to *not* fsync() on all Append() calls
I20260929 16:47:34.369894 23695 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "0d87403c02d94935bdaa198173c21cc1"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44646
I20260929 16:47:34.372236 23870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45f02b7b782346e58158981174a8c839: No bootstrap required, opened a new log
I20260929 16:47:34.376019 23720 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d87403c02d94935bdaa198173c21cc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d87403c02d94935bdaa198173c21cc1" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 35461 } } peers { permanent_uuid: "f980ff460ed643cab40f4a477ac52c43" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 44579 } } peers { permanent_uuid: "45f02b7b782346e58158981174a8c839" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43445 } }
I20260929 16:47:34.376410 23720 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d87403c02d94935bdaa198173c21cc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d87403c02d94935bdaa198173c21cc1, State: Initialized, Role: FOLLOWER
I20260929 16:47:34.377015 23720 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d87403c02d94935bdaa198173c21cc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d87403c02d94935bdaa198173c21cc1" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 35461 } } peers { permanent_uuid: "f980ff460ed643cab40f4a477ac52c43" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 44579 } } peers { permanent_uuid: "45f02b7b782346e58158981174a8c839" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43445 } }
I20260929 16:47:34.378185 23879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d87403c02d94935bdaa198173c21cc1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d87403c02d94935bdaa198173c21cc1" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 35461 } } peers { permanent_uuid: "f980ff460ed643cab40f4a477ac52c43" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 44579 } } peers { permanent_uuid: "45f02b7b782346e58158981174a8c839" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43445 } } }
I20260929 16:47:34.378737 23879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d87403c02d94935bdaa198173c21cc1 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:47:34.379633 23720 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d87403c02d94935bdaa198173c21cc1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:47:34.379678 23695 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d87403c02d94935bdaa198173c21cc1 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 16:47:34.379902 23695 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d87403c02d94935bdaa198173c21cc1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:47:34.382928 23870 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45f02b7b782346e58158981174a8c839 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d87403c02d94935bdaa198173c21cc1" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 35461 } } peers { permanent_uuid: "f980ff460ed643cab40f4a477ac52c43" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 44579 } } peers { permanent_uuid: "45f02b7b782346e58158981174a8c839" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43445 } }
I20260929 16:47:34.383410 23870 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45f02b7b782346e58158981174a8c839 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45f02b7b782346e58158981174a8c839, State: Initialized, Role: FOLLOWER
I20260929 16:47:34.383920 23870 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45f02b7b782346e58158981174a8c839 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d87403c02d94935bdaa198173c21cc1" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 35461 } } peers { permanent_uuid: "f980ff460ed643cab40f4a477ac52c43" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 44579 } } peers { permanent_uuid: "45f02b7b782346e58158981174a8c839" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43445 } }
I20260929 16:47:34.384601 23695 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d87403c02d94935bdaa198173c21cc1 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d87403c02d94935bdaa198173c21cc1" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 35461 } } peers { permanent_uuid: "f980ff460ed643cab40f4a477ac52c43" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 44579 } } peers { permanent_uuid: "45f02b7b782346e58158981174a8c839" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43445 } }
I20260929 16:47:34.384882 23882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45f02b7b782346e58158981174a8c839 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d87403c02d94935bdaa198173c21cc1" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 35461 } } peers { permanent_uuid: "f980ff460ed643cab40f4a477ac52c43" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 44579 } } peers { permanent_uuid: "45f02b7b782346e58158981174a8c839" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43445 } } }
I20260929 16:47:34.385411 23882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45f02b7b782346e58158981174a8c839 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:47:34.386178 23870 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45f02b7b782346e58158981174a8c839 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:47:34.387468 23695 leader_election.cc:288] T 00000000000000000000000000000000 P 0d87403c02d94935bdaa198173c21cc1 [CANDIDATE]: Term 1 election: Requested vote from peers f980ff460ed643cab40f4a477ac52c43 (127.0.197.125:44579), 45f02b7b782346e58158981174a8c839 (127.0.197.124:43445)
I20260929 16:47:34.388281 23845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0d87403c02d94935bdaa198173c21cc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "45f02b7b782346e58158981174a8c839"
I20260929 16:47:34.388893 23845 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45f02b7b782346e58158981174a8c839 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:47:34.394156 23845 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 45f02b7b782346e58158981174a8c839 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d87403c02d94935bdaa198173c21cc1 in term 1.
I20260929 16:47:34.395637 23669 leader_election.cc:302] T 00000000000000000000000000000000 P 0d87403c02d94935bdaa198173c21cc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d87403c02d94935bdaa198173c21cc1, 45f02b7b782346e58158981174a8c839; no voters:
I20260929 16:47:34.396406 23879 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d87403c02d94935bdaa198173c21cc1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:47:34.397181 23879 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d87403c02d94935bdaa198173c21cc1 [term 1 LEADER]: Becoming Leader. State: Replica: 0d87403c02d94935bdaa198173c21cc1, State: Running, Role: LEADER
I20260929 16:47:34.397845 23879 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d87403c02d94935bdaa198173c21cc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d87403c02d94935bdaa198173c21cc1" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 35461 } } peers { permanent_uuid: "f980ff460ed643cab40f4a477ac52c43" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 44579 } } peers { permanent_uuid: "45f02b7b782346e58158981174a8c839" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43445 } }
W20260929 16:47:34.400944 23674 tablet.cc:2369] T 00000000000000000000000000000000 P 0d87403c02d94935bdaa198173c21cc1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:47:34.411571 23885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d87403c02d94935bdaa198173c21cc1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d87403c02d94935bdaa198173c21cc1. Latest consensus state: current_term: 1 leader_uuid: "0d87403c02d94935bdaa198173c21cc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d87403c02d94935bdaa198173c21cc1" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 35461 } } peers { permanent_uuid: "f980ff460ed643cab40f4a477ac52c43" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 44579 } } peers { permanent_uuid: "45f02b7b782346e58158981174a8c839" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43445 } } }
I20260929 16:47:34.412177 23885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d87403c02d94935bdaa198173c21cc1 [sys.catalog]: This master's current role is: LEADER
I20260929 16:47:34.426399 23890 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:47:34.432497 23890 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:47:34.445326 23794 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43445 } attempt: 5
Sep 29 16:47:34 dist-test-slave-l31f krb5kdc[23640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790700453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.197.125@KRBTEST.COM for kudu/127.0.197.124@KRBTEST.COM
I20260929 16:47:34.461082 23845 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 45f02b7b782346e58158981174a8c839 [term 1 FOLLOWER]: Refusing update from remote peer 0d87403c02d94935bdaa198173c21cc1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:47:34.463238 23912 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 16:47:34.463371 23885 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0d87403c02d94935bdaa198173c21cc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45f02b7b782346e58158981174a8c839" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:47:34.465878 23802 master_runner.cc:186] Error getting master registration for 127.0.197.126:35461: OK, instance_id { permanent_uuid: "0d87403c02d94935bdaa198173c21cc1" instance_seqno: 1790700453162649 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 16:47:34.482079 23802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 16:47:34.471472 23653 master_runner.cc:186] Error getting master registration for 127.0.197.125:44579: OK, instance_id { permanent_uuid: "f980ff460ed643cab40f4a477ac52c43" instance_seqno: 1790700453710638 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 16:47:34.483175 23653 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 16:47:34.487458 23803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:34.446131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.197.124:43445 (local address 127.0.0.1:41112)
0929 16:47:34.446600 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0929 16:47:34.446623 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 16:47:34.446778 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:47:34.447641 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:47:34.447647 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:47:34.448301 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:47:34.448576 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:47:34.448593 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:47:34.450698 (+ 2105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:47:34.450707 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:47:34.452584 (+ 1877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:47:34.452607 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:47:34.454165 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:47:34.454174 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:47:34.454572 (+ 398us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:34.455607 (+ 1035us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:47:34.455652 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:47:34.468660 (+ 13008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:47:34.476162 (+ 7502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:47:34.476169 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:47:34.476173 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:47:34.477318 (+ 1145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:47:34.478512 (+ 1194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:47:34.478518 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:47:34.478521 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:47:34.478937 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:47:34.481681 (+ 2744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:47:34.481690 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:47:34.482518 (+ 828us) client_negotiation.cc:769] Sending connection context
0929 16:47:34.486540 (+ 4022us) client_negotiation.cc:240] Negotiation successful
0929 16:47:34.486991 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289}
I20260929 16:47:34.488628 23876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:34.447077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41112 (local address 127.0.197.124:43445)
0929 16:47:34.447238 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:34.447243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:34.447263 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:47:34.447385 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:34.447390 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:34.447492 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:34.447601 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:34.449018 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:34.450438 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:34.452878 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:34.453684 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:34.453813 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:34.469009 (+ 15196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:47:34.469017 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:47:34.469022 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:47:34.469074 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:47:34.475811 (+ 6737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:34.477553 (+ 1742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:34.477596 (+ 43us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:34.477602 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:34.478114 (+ 512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:34.479241 (+ 1127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:34.479251 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:34.479292 (+ 41us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:34.480833 (+ 1541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:47:34.483442 (+ 2609us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:34.486995 (+ 3553us) server_negotiation.cc:299] Negotiation successful
0929 16:47:34.487726 (+ 731us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 16:47:34.495078 23794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f980ff460ed643cab40f4a477ac52c43: Bootstrap starting.
I20260929 16:47:34.499600 23794 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f980ff460ed643cab40f4a477ac52c43: Neither blocks nor log segments found. Creating new log.
I20260929 16:47:34.501031 23794 log.cc:824] T 00000000000000000000000000000000 P f980ff460ed643cab40f4a477ac52c43: Log is configured to *not* fsync() on all Append() calls
I20260929 16:47:34.502871 23882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45f02b7b782346e58158981174a8c839 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d87403c02d94935bdaa198173c21cc1. Latest consensus state: current_term: 1 leader_uuid: "0d87403c02d94935bdaa198173c21cc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d87403c02d94935bdaa198173c21cc1" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 35461 } } peers { permanent_uuid: "f980ff460ed643cab40f4a477ac52c43" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 44579 } } peers { permanent_uuid: "45f02b7b782346e58158981174a8c839" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43445 } } }
I20260929 16:47:34.503245 23882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45f02b7b782346e58158981174a8c839 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:47:34.504719 23873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:34.449399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41126 (local address 127.0.197.124:43445)
0929 16:47:34.449634 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:34.449649 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:34.459565 (+ 9916us) server_negotiation.cc:407] Connection header received
0929 16:47:34.460127 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:34.460136 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:34.460253 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:34.460345 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:34.468766 (+ 8421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:34.469924 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:34.476298 (+ 6374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:34.477155 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:34.477273 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:34.488634 (+ 11361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:47:34.488642 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:47:34.488647 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:47:34.488696 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:47:34.492861 (+ 4165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:34.495276 (+ 2415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:34.495285 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:34.495290 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:34.495480 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:34.502574 (+ 7094us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:34.502581 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:34.502585 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:34.503137 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:47:34.503453 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:34.504055 (+ 602us) server_negotiation.cc:299] Negotiation successful
0929 16:47:34.504336 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 16:47:34.508380 23794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f980ff460ed643cab40f4a477ac52c43: No bootstrap required, opened a new log
I20260929 16:47:34.512053 23874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:34.457293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44650 (local address 127.0.197.126:35461)
0929 16:47:34.457598 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:34.457608 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:34.463853 (+ 6245us) server_negotiation.cc:407] Connection header received
0929 16:47:34.464164 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:34.464172 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:34.464286 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:34.464392 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:34.472197 (+ 7805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:34.473442 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:34.480187 (+ 6745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:34.480927 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:34.481046 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:34.496073 (+ 15027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:47:34.496080 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:47:34.496086 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:47:34.496134 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:47:34.500039 (+ 3905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:34.502581 (+ 2542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:34.502587 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:34.502590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:34.502989 (+ 399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:34.508410 (+ 5421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:34.508419 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:34.508426 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:34.508978 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:47:34.509331 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:34.511145 (+ 1814us) server_negotiation.cc:299] Negotiation successful
0929 16:47:34.511515 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260929 16:47:34.515231 23915 mvcc.cc:205] Tried to move back new op lower bound from 7334709061460197376 to 7334709061221765120. Current Snapshot: MvccSnapshot[applied={T|T < 7334709061460197376}]
I20260929 16:47:34.516063 23882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45f02b7b782346e58158981174a8c839 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0d87403c02d94935bdaa198173c21cc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d87403c02d94935bdaa198173c21cc1" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 35461 } } peers { permanent_uuid: "f980ff460ed643cab40f4a477ac52c43" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 44579 } } peers { permanent_uuid: "45f02b7b782346e58158981174a8c839" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43445 } } }
I20260929 16:47:34.516335 23882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45f02b7b782346e58158981174a8c839 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:47:34.521129 23885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d87403c02d94935bdaa198173c21cc1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0d87403c02d94935bdaa198173c21cc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d87403c02d94935bdaa198173c21cc1" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 35461 } } peers { permanent_uuid: "f980ff460ed643cab40f4a477ac52c43" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 44579 } } peers { permanent_uuid: "45f02b7b782346e58158981174a8c839" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43445 } } }
I20260929 16:47:34.521418 23885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d87403c02d94935bdaa198173c21cc1 [sys.catalog]: This master's current role is: LEADER
I20260929 16:47:34.522740 23890 catalog_manager.cc:1347] Generated new cluster ID: a7961aaebe114441a59cdab30593b432
I20260929 16:47:34.522876 23890 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:47:34.526640 23799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:34.468269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51008 (local address 127.0.197.125:44579)
0929 16:47:34.468648 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:34.468656 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:34.479205 (+ 10549us) server_negotiation.cc:407] Connection header received
0929 16:47:34.482073 (+ 2868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:34.482079 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:34.482178 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:34.482257 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:34.495234 (+ 12977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:34.496472 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:34.506890 (+ 10418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:34.507967 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:34.508113 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:34.517728 (+ 9615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:47:34.517741 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:47:34.517749 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:47:34.517812 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:47:34.522405 (+ 4593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:34.523890 (+ 1485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:34.523897 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:34.523901 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:34.524107 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:47:34.524599 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:47:34.524604 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:47:34.524607 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:47:34.525061 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:47:34.525329 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:34.525937 (+ 608us) server_negotiation.cc:299] Negotiation successful
0929 16:47:34.526219 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238}
I20260929 16:47:34.527340 23794 raft_consensus.cc:348] T 00000000000000000000000000000000 P f980ff460ed643cab40f4a477ac52c43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d87403c02d94935bdaa198173c21cc1" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 35461 } } peers { permanent_uuid: "f980ff460ed643cab40f4a477ac52c43" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 44579 } } peers { permanent_uuid: "45f02b7b782346e58158981174a8c839" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43445 } }
I20260929 16:47:34.528223 23794 raft_consensus.cc:729] T 00000000000000000000000000000000 P f980ff460ed643cab40f4a477ac52c43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f980ff460ed643cab40f4a477ac52c43, State: Initialized, Role: FOLLOWER
I20260929 16:47:34.529376 23794 consensus_queue.cc:260] T 00000000000000000000000000000000 P f980ff460ed643cab40f4a477ac52c43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d87403c02d94935bdaa198173c21cc1" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 35461 } } peers { permanent_uuid: "f980ff460ed643cab40f4a477ac52c43" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 44579 } } peers { permanent_uuid: "45f02b7b782346e58158981174a8c839" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43445 } }
I20260929 16:47:34.532291 23794 sys_catalog.cc:564] T 00000000000000000000000000000000 P f980ff460ed643cab40f4a477ac52c43 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:47:34.532366 23769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0d87403c02d94935bdaa198173c21cc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f980ff460ed643cab40f4a477ac52c43"
I20260929 16:47:34.532850 23769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f980ff460ed643cab40f4a477ac52c43 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:47:34.537514 23769 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f980ff460ed643cab40f4a477ac52c43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d87403c02d94935bdaa198173c21cc1 in term 1.
I20260929 16:47:34.538829 23768 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f980ff460ed643cab40f4a477ac52c43 [term 1 FOLLOWER]: Refusing update from remote peer 0d87403c02d94935bdaa198173c21cc1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:47:34.539930 23879 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0d87403c02d94935bdaa198173c21cc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f980ff460ed643cab40f4a477ac52c43" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:47:34.539865 23920 sys_catalog.cc:455] T 00000000000000000000000000000000 P f980ff460ed643cab40f4a477ac52c43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d87403c02d94935bdaa198173c21cc1" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 35461 } } peers { permanent_uuid: "f980ff460ed643cab40f4a477ac52c43" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 44579 } } peers { permanent_uuid: "45f02b7b782346e58158981174a8c839" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43445 } } }
I20260929 16:47:34.540323 23920 sys_catalog.cc:458] T 00000000000000000000000000000000 P f980ff460ed643cab40f4a477ac52c43 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:47:34.572899 23890 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:47:34.574997 23890 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:47:34.585489 23920 sys_catalog.cc:455] T 00000000000000000000000000000000 P f980ff460ed643cab40f4a477ac52c43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d87403c02d94935bdaa198173c21cc1. Latest consensus state: current_term: 1 leader_uuid: "0d87403c02d94935bdaa198173c21cc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d87403c02d94935bdaa198173c21cc1" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 35461 } } peers { permanent_uuid: "f980ff460ed643cab40f4a477ac52c43" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 44579 } } peers { permanent_uuid: "45f02b7b782346e58158981174a8c839" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43445 } } }
I20260929 16:47:34.585989 23920 sys_catalog.cc:458] T 00000000000000000000000000000000 P f980ff460ed643cab40f4a477ac52c43 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:47:34.586544 23928 sys_catalog.cc:455] T 00000000000000000000000000000000 P f980ff460ed643cab40f4a477ac52c43 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0d87403c02d94935bdaa198173c21cc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d87403c02d94935bdaa198173c21cc1" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 35461 } } peers { permanent_uuid: "f980ff460ed643cab40f4a477ac52c43" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 44579 } } peers { permanent_uuid: "45f02b7b782346e58158981174a8c839" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43445 } } }
I20260929 16:47:34.587114 23928 sys_catalog.cc:458] T 00000000000000000000000000000000 P f980ff460ed643cab40f4a477ac52c43 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:47:34.588908 23879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d87403c02d94935bdaa198173c21cc1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0d87403c02d94935bdaa198173c21cc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d87403c02d94935bdaa198173c21cc1" member_type: VOTER last_known_addr { host: "127.0.197.126" port: 35461 } } peers { permanent_uuid: "f980ff460ed643cab40f4a477ac52c43" member_type: VOTER last_known_addr { host: "127.0.197.125" port: 44579 } } peers { permanent_uuid: "45f02b7b782346e58158981174a8c839" member_type: VOTER last_known_addr { host: "127.0.197.124" port: 43445 } } }
I20260929 16:47:34.589319 23879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d87403c02d94935bdaa198173c21cc1 [sys.catalog]: This master's current role is: LEADER
I20260929 16:47:34.606417 23890 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d87403c02d94935bdaa198173c21cc1: Generated new TSK 0
I20260929 16:47:34.611824 23890 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:47:34.618129 23937 catalog_manager.cc:1259] Loaded cluster ID: a7961aaebe114441a59cdab30593b432
I20260929 16:47:34.618259 23937 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f980ff460ed643cab40f4a477ac52c43: loading cluster ID for follower catalog manager: success
I20260929 16:47:34.622274 23937 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f980ff460ed643cab40f4a477ac52c43: acquiring CA information for follower catalog manager: success
I20260929 16:47:34.625226 23937 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f980ff460ed643cab40f4a477ac52c43: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 16:47:34.705991 23873 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:47:34.698143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41134 (local address 127.0.197.124:43445)
0929 16:47:34.698333 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:34.698340 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:34.703721 (+ 5381us) server_negotiation.cc:407] Connection header received
0929 16:47:34.704055 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:34.704062 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:34.704159 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:34.704243 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:34.705640 (+ 1397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41134: BlockingRecv error: recv got EOF from 127.0.0.1:41134 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260929 16:47:34.708895 23874 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:47:34.697589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44666 (local address 127.0.197.126:35461)
0929 16:47:34.697826 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:34.697833 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:34.707042 (+ 9209us) server_negotiation.cc:407] Connection header received
0929 16:47:34.707337 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:34.707344 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:34.707442 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:34.707532 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:34.708620 (+ 1088us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44666: BlockingRecv error: recv got EOF from 127.0.0.1:44666 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260929 16:47:34.712008 23799 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:47:34.709936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51020 (local address 127.0.197.125:44579)
0929 16:47:34.710152 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:34.710159 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:34.710277 (+ 118us) server_negotiation.cc:407] Connection header received
0929 16:47:34.710463 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:34.710469 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:34.710568 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:34.710686 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:34.711721 (+ 1035us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51020: BlockingRecv error: recv got EOF from 127.0.0.1:51020 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 16:47:34.748837 23799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:34.715373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51028 (local address 127.0.197.125:44579)
0929 16:47:34.715559 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:34.715572 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:34.739641 (+ 24069us) server_negotiation.cc:407] Connection header received
0929 16:47:34.739883 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:34.739891 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:34.740146 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:34.740350 (+ 204us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:47:34.741137 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:34.742493 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:34.744983 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:34.745983 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:34.746149 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:34.747084 (+ 935us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:47:34.747895 (+ 811us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:47:34.748031 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:34.748317 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 16:47:34.748437 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260929 16:47:34.760728 23873 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:47:34.757385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41142 (local address 127.0.197.124:43445)
0929 16:47:34.757591 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:34.757597 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:34.758129 (+ 532us) server_negotiation.cc:407] Connection header received
0929 16:47:34.758500 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:34.758507 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:34.758645 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:34.758730 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:34.760475 (+ 1745us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41142: BlockingRecv error: recv got EOF from 127.0.0.1:41142 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 16:47:34.775164 23799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:34.766689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51042 (local address 127.0.197.125:44579)
0929 16:47:34.766912 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:34.766919 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:34.767464 (+ 545us) server_negotiation.cc:407] Connection header received
0929 16:47:34.767809 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:34.767816 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:34.767916 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:34.767988 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:47:34.769045 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:34.770296 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:34.772685 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:34.773487 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:34.773614 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:34.774204 (+ 590us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:47:34.774420 (+ 216us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:47:34.774531 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:34.774712 (+ 181us) server_negotiation.cc:299] Negotiation successful
0929 16:47:34.774812 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260929 16:47:34.786437 23873 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:47:34.783504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41154 (local address 127.0.197.124:43445)
0929 16:47:34.783730 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:34.783736 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:34.784346 (+ 610us) server_negotiation.cc:407] Connection header received
0929 16:47:34.784691 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:34.784698 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:34.784808 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:34.784889 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:34.786141 (+ 1252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41154: BlockingRecv error: recv got EOF from 127.0.0.1:41154 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 16:47:34.800899 23799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:34.793146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51044 (local address 127.0.197.125:44579)
0929 16:47:34.793362 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:34.793369 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:34.793952 (+ 583us) server_negotiation.cc:407] Connection header received
0929 16:47:34.794173 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:34.794181 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:34.794284 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:34.794402 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:47:34.794919 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:34.796207 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:34.798363 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:34.799205 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:34.799366 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:34.799895 (+ 529us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:47:34.800130 (+ 235us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:47:34.800229 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:34.800418 (+ 189us) server_negotiation.cc:299] Negotiation successful
0929 16:47:34.800541 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260929 16:47:34.810706 23873 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:47:34.808328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41170 (local address 127.0.197.124:43445)
0929 16:47:34.808523 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:34.808529 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:34.809056 (+ 527us) server_negotiation.cc:407] Connection header received
0929 16:47:34.809258 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:34.809265 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:34.809372 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:34.809500 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:34.810424 (+ 924us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41170: BlockingRecv error: recv got EOF from 127.0.0.1:41170 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 16:47:34.826025 23799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:34.818491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51050 (local address 127.0.197.125:44579)
0929 16:47:34.818696 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:34.818703 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:34.819260 (+ 557us) server_negotiation.cc:407] Connection header received
0929 16:47:34.819483 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:34.819495 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:34.819583 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:34.819682 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:47:34.820168 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:34.821293 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:34.823250 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:34.824174 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:34.824340 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:34.824981 (+ 641us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:47:34.825148 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:47:34.825252 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:34.825444 (+ 192us) server_negotiation.cc:299] Negotiation successful
0929 16:47:34.825552 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260929 16:47:34.845348 23873 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:47:34.834052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41182 (local address 127.0.197.124:43445)
0929 16:47:34.834387 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:34.834395 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:34.843355 (+ 8960us) server_negotiation.cc:407] Connection header received
0929 16:47:34.843642 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:34.843650 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:34.843768 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:34.843863 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:34.845090 (+ 1227us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41182: BlockingRecv error: recv got EOF from 127.0.0.1:41182 (error 108)
Metrics: {"server-negotiator.queue_time_us":180}
I20260929 16:47:34.866667 23799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:34.857249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51066 (local address 127.0.197.125:44579)
0929 16:47:34.857447 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:34.857454 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:34.858709 (+ 1255us) server_negotiation.cc:407] Connection header received
0929 16:47:34.859220 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:34.859227 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:34.859325 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:34.859394 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:47:34.860134 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:34.861308 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:34.864025 (+ 2717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:34.864822 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:34.864948 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:34.865740 (+ 792us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:47:34.865896 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:47:34.866028 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:34.866223 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 16:47:34.866319 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260929 16:47:34.876821 23873 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:47:34.874228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41188 (local address 127.0.197.124:43445)
0929 16:47:34.874436 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:34.874443 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:34.875069 (+ 626us) server_negotiation.cc:407] Connection header received
0929 16:47:34.875289 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:34.875296 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:34.875404 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:34.875522 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:34.876546 (+ 1024us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41188: BlockingRecv error: recv got EOF from 127.0.0.1:41188 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 16:47:34.904453 23799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:34.897604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51076 (local address 127.0.197.125:44579)
0929 16:47:34.897821 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:34.897827 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:34.898377 (+ 550us) server_negotiation.cc:407] Connection header received
0929 16:47:34.898719 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:34.898729 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:34.898841 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:34.898924 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:47:34.899650 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:34.900845 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:34.902343 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:34.902949 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:34.903203 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:34.903663 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:47:34.903803 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:47:34.903907 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:34.904064 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 16:47:34.904160 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260929 16:47:34.913129 23873 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:47:34.910763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41196 (local address 127.0.197.124:43445)
0929 16:47:34.911047 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:34.911057 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:34.911616 (+ 559us) server_negotiation.cc:407] Connection header received
0929 16:47:34.911840 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:34.911846 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:34.911948 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:34.912065 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:34.912893 (+ 828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41196: BlockingRecv error: recv got EOF from 127.0.0.1:41196 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 16:47:34.960126 23799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:34.952413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51090 (local address 127.0.197.125:44579)
0929 16:47:34.952642 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:34.952651 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:34.953113 (+ 462us) server_negotiation.cc:407] Connection header received
0929 16:47:34.953361 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:34.953367 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:34.953478 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:34.953600 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:47:34.954145 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:34.955435 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:34.957542 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:34.958367 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:34.958525 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:34.959138 (+ 613us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:47:34.959310 (+ 172us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:47:34.959428 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:34.959636 (+ 208us) server_negotiation.cc:299] Negotiation successful
0929 16:47:34.959727 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260929 16:47:34.971705 23873 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:47:34.969464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41204 (local address 127.0.197.124:43445)
0929 16:47:34.969662 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:34.969670 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:34.969786 (+ 116us) server_negotiation.cc:407] Connection header received
0929 16:47:34.970144 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:34.970151 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:34.970261 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:34.970387 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:34.971437 (+ 1050us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41204: BlockingRecv error: recv got EOF from 127.0.0.1:41204 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 16:47:35.052191 23799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:35.044831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51100 (local address 127.0.197.125:44579)
0929 16:47:35.045037 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:35.045044 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:35.045617 (+ 573us) server_negotiation.cc:407] Connection header received
0929 16:47:35.045819 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:35.045826 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:35.045918 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:35.046031 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:47:35.046518 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:35.047716 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:35.049829 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:35.050663 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:35.050796 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:35.051373 (+ 577us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:47:35.051546 (+ 173us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:47:35.051638 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:35.051799 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 16:47:35.051899 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260929 16:47:35.065984 23873 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:47:35.063172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41216 (local address 127.0.197.124:43445)
0929 16:47:35.063364 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:35.063371 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:35.064306 (+ 935us) server_negotiation.cc:407] Connection header received
0929 16:47:35.064505 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:35.064511 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:35.064636 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:35.064756 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:35.065692 (+ 936us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41216: BlockingRecv error: recv got EOF from 127.0.0.1:41216 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 16:47:35.202849 23799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:35.197412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51110 (local address 127.0.197.125:44579)
0929 16:47:35.197593 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:35.197598 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:35.198027 (+ 429us) server_negotiation.cc:407] Connection header received
0929 16:47:35.198300 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:35.198304 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:35.198433 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:35.198546 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:47:35.199038 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:35.199768 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:35.201108 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:35.201544 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:35.201699 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:35.202103 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:47:35.202217 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:47:35.202319 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:35.202468 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 16:47:35.202554 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260929 16:47:35.211371 23873 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:47:35.209215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41232 (local address 127.0.197.124:43445)
0929 16:47:35.209466 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:35.209472 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:35.209871 (+ 399us) server_negotiation.cc:407] Connection header received
0929 16:47:35.210094 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:35.210099 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:35.210180 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:35.210294 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:47:35.211126 (+ 832us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41232: BlockingRecv error: recv got EOF from 127.0.0.1:41232 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
I20260929 16:47:35.468385 23912 catalog_manager.cc:1259] Loaded cluster ID: a7961aaebe114441a59cdab30593b432
I20260929 16:47:35.468575 23912 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 45f02b7b782346e58158981174a8c839: loading cluster ID for follower catalog manager: success
I20260929 16:47:35.474694 23912 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 45f02b7b782346e58158981174a8c839: acquiring CA information for follower catalog manager: success
I20260929 16:47:35.477998 23912 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 45f02b7b782346e58158981174a8c839: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 16:47:35.483824 23799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:35.476779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51124 (local address 127.0.197.125:44579)
0929 16:47:35.476998 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:35.477004 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:35.477589 (+ 585us) server_negotiation.cc:407] Connection header received
0929 16:47:35.477871 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:35.477878 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:35.478012 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:35.478125 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:47:35.478629 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:35.479829 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:35.481378 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:35.482178 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:35.482342 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:35.482935 (+ 593us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:47:35.483136 (+ 201us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:47:35.483236 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:35.483429 (+ 193us) server_negotiation.cc:299] Negotiation successful
0929 16:47:35.483527 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260929 16:47:35.499300 23873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:35.491284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41244 (local address 127.0.197.124:43445)
0929 16:47:35.491519 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:35.491526 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:35.491858 (+ 332us) server_negotiation.cc:407] Connection header received
0929 16:47:35.492144 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:35.492149 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:35.492365 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:35.492488 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:47:35.493173 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:35.494026 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:35.496229 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:35.496694 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:35.496814 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:35.497683 (+ 869us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:47:35.498390 (+ 707us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:47:35.498526 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:35.498743 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 16:47:35.498836 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260929 16:47:35.527740 23873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:35.506807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41250 (local address 127.0.197.124:43445)
0929 16:47:35.508031 (+ 1224us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:35.508038 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:35.519279 (+ 11241us) server_negotiation.cc:407] Connection header received
0929 16:47:35.519592 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:35.519599 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:35.519699 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:35.519779 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:47:35.520606 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:35.521834 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:35.524879 (+ 3045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:35.525644 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:35.525753 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:35.526621 (+ 868us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:47:35.526768 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:47:35.526846 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:35.527262 (+ 416us) server_negotiation.cc:299] Negotiation successful
0929 16:47:35.527358 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1064}
I20260929 16:47:35.531975 24052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:35.506197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44674 (local address 127.0.197.126:35461)
0929 16:47:35.521563 (+ 15366us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:35.521573 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:35.521602 (+ 29us) server_negotiation.cc:407] Connection header received
0929 16:47:35.521759 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:35.521769 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:35.522090 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:35.522356 (+ 266us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:47:35.523037 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:35.526445 (+ 3408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:35.529321 (+ 2876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:35.530032 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:35.530167 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:35.530768 (+ 601us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:47:35.531267 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:47:35.531405 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:35.531560 (+ 155us) server_negotiation.cc:299] Negotiation successful
0929 16:47:35.531635 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15188,"thread_start_us":16298,"threads_started":1}
I20260929 16:47:35.532397 23799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:47:35.519614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51138 (local address 127.0.197.125:44579)
0929 16:47:35.519861 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0929 16:47:35.519871 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:47:35.524050 (+ 4179us) server_negotiation.cc:407] Connection header received
0929 16:47:35.524355 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:47:35.524361 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:47:35.524480 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:47:35.524556 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:47:35.525390 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:35.526445 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:35.529354 (+ 2909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:47:35.530032 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:47:35.530167 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:47:35.531158 (+ 991us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:47:35.531290 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:47:35.531388 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 16:47:35.531975 (+ 587us) server_negotiation.cc:299] Negotiation successful
0929 16:47:35.532061 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 16:47:35.537117 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 23653
I20260929 16:47:35.538266 23715 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:47:35.680562 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 23653
W20260929 16:47:35.711201 23743 connection.cc:535] server connection from 127.0.0.1:34506 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 16:47:35.711416 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 23727
I20260929 16:47:35.712613 23789 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:47:35.824609 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 23727
I20260929 16:47:35.849488 789 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu pid 23802
I20260929 16:47:35.850716 23865 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:47:35.978886 789 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQmsv3Z/build/asan/bin/kudu with pid 23802
2026-09-29T16:47:35Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4697 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9891 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (387677 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST